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The Meaning of “Paradox” 


What is meant by the expression “paradox” in “Paradoxes of Interactivity”? 
“Paradox” as used in the title of the book refers to the ordinary meaning of the 
word, and not to the well-known paradoxes of logic and mathematics such 
as Russell’s! set-theoretical paradox or Zeno’s? paradoxes of plurality and 
motion. The semantic field of the ordinary meaning of the word “paradox” 
derives from the ancient Greek word “paradoxos” consisting of “para” mean- 
ing “contrary” and “döxa” meaning “opinion”. In Book V of his “Republic” 
Plato? speaks of “paradoxos logos”. Used in this sense the meaning of “para- 


“ 


dox” is “a statement contrary to expectation”, “an incredible statement”, “a 
statement contrary to accepted opinion”, “against common sense or ordinary 
opinion”, “provocative to accepted opinion or common sense”, “contrary to 
generally accepted belief” or “something surprising”. What are the provoca- 


tive or incredible ideas associated with “interactivity”? 


Interactivity: A Semantic Field 


In general, an explanation of “interactivity” or “interaction” refers back 
to “action”, and in the social sciences action is presupposed to depend on 
an active human subject intentionally acting upon an object or another 
subject. Interaction only takes place between humans, because objects, 
like machines, are incapable of intentionality. In the case of human action 
humans are ascribed agency. In sociology “agency” is often contrasted with 
“structure”. But actually, structure is both an outcome of previous agency 
and a constraint upon it. Two semantic fields can be associated with “agency”. 
First, in the social sciences the sense of “agency” is mainly judicial, political 
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or economical. This meaning is related to authority and assignment of power & s 
or official duties to humans, e.g. “assignee” or “agent”. The second mean- ae 
ing of “agency” and “agent” is mostly found in the natural sciences such as À 
chemistry, biology, and physics. It is associated with effect, tool, activity, e.g. Fd 
“protective agent”. To summarise: There are two fields of meanings concern- 4 j 
ing “agency”, “agent”, “action”, “interaction” and “to act upon”. One semantic Pa 
field concerns the social sciences, and strongly relates to the idea of an inten- 

tional being, a human. The other concerns the natural sciences and refers to 

the idea of effect. 

1 Russell 1903, 8 101 Á 
2 Salmon 2001 Ef 
3 Plato’s “Republic“ 472 a6; see Liddell/Scott 1996, p. 1309 F 

a 
S mn 


With the advent of computational technology and systems the situa- 
tion has changed: machines are attributed the active role in human’s use 
of machines. They become subjects of actions and agents. One speaks of 
humans interacting with computers. 

Agent technology and social robotics are two important recent examples 
to illustrate that the difference between humans and machines is becoming 
increasingly blurred. The meaning of “agent” applied in the social sciences 
only to human agents is being transferred to software: Special tasks such 
as internet searches and communication on behalf of a human are assigned 
to personal software agents. In entertainment, therapy, e.g. autism therapy, 
and at home, e.g. as a robot companion, robots are interacting with humans. 
Furthermore, ubiquitous computing seems to make the computer “disap- 
pear” and at the same time be more and more entangled in day-to-day life: 
Things are beginning to talk.” 

Furthermore, the asymmetric relation of human’s use of machines is 
becoming symmetric. Machines are becoming the subject of interaction. 
Humans are interacting with machines, machines are interacting with 
machines and humans are interacting with humans via machines. In gen- 
eral, there are hybrid networks consisting of human and machine interact- 
ing with each other. The use of “interactivity” suggests that the difference 
between humans and machines evaporates. This is the main paradox associ- 
ated with “interactivity”! 


Humans and Machines: An Evolving Discontinuity 


“Interactivity” indicates that at present a phase of fundamental change is 
being undergone in the ontological difference between humans and machines: 
This discontinuity is beginning to disappear. This important insight has been 
realised but articulated differently by many authors from different disci- 
plines.° Especially Bruce Mazlish developed this insight further and pointed 
out that in order to cope with emerging social and cultural problems humans 
must accept the continuity of humans and machines. His thesis is that “man 
is breaking past the discontinuity between himself and machines.” Mazlish 


argues: Man is now becoming aware that “his own evolution is inextricably 


A 


4 Payr/Trappl 2004; Dautenhahn 2002 


5 For instane, O’Sullivan/Igoe 2004 and Igoe 2007 introduce into physical computing 
in media art and design using Arduino, Processing and other “tools“. 


6 E.g. Robertson 1998, 2003; Ford/Glymour/Hayes 2006; Hubig/Koslowski 2008; 
Mazlish 1967, 1993 


7 Mazlish 1967, p. 14; Mazlish 1993 elaborates the fourth discontinuity thesis. 


10 


interwoven with his use and development of tools”, and that “the same scien- 
tific concepts help explain the workings of himself and of his machines.” 

At present the strongest scientific thesis in this sense is put forward by 
cognitive scientists: They argue that human beings are (logical) automata or 
that all natural human functions are best explained by (finite) automata in 
the sense of automata theory.'° 

Mazlish’s claim is strongly supported by the emergence of new scientific 
disciplines, subdisciplines and research areas as well as art forms and cul- 
tural applications of computing, such as, to mention just a few, cognitive 
science, computational and cognitive neuroscience, techno- and biosciences, 
ubiquitous, physical and art computing, social and educational robotics, 
neuro-robotics, human-computer and human-robot interaction, interaction 
design, and interactive and new media art. 


“Action” and “Interaction”: Some Definitions 


These developments require the social sciences, especially sociology, to 
interpret “interaction”, “interactivity”, “agency” and “agent” in the same man- 
ner as the natural sciences. Mario Bunge developed definitions of these terms 
whose meanings encompass both the social and the natural sciences.'! 

He considers “action” as a general (ontological) concept. The general 
idea captured in formalising the action relation is “What one thing does to 
another.”’? In his formalisation’? Bunge uses concepts from set theory in 
order to define the action relation “x acts upon y” or “the action that thing x 
exerts on thing y”. The expression “x acts upon y” is defined as a set-theoretic 
difference or relative complement of the history of y in the presence ofx, and 
the history of y in the absence of x. The history of an object x is formed by 
the values v of its state function F for all time points t over a time period T. 
A state function F can be conceived of as a list of all known properties for 
some kind of objects. The concept of “interaction” is based on the definition 
of “to act upon”: Two different things interact if and only if each acts upon 
the other. Human action appears as a special case of action. An action is a 
human action if and only if at least the agent of the action relation is a per- 
8 Mazlish 1967, p. 14 
9 Mazlish 1967, p. 14 
10 E.g. Boden 2007; Burks 1972-73, 1990; Nelson 1988. Concerning the terms 
“machine” and “finite automata“ see Minsky 1972, pp. 1-7 and pp. 11-31. 

11 Bunge 1998, 2003 
12 Bunge 2003, p. 9 
13 Rudolf Kaehr’s contribution takes a formalised approach to interaction. Kaehr’s 


contribution should be viewed in the context of Gotthard Günther’s ideas concerning the 
formalisation of the ideas of an “objective spirit/mind” and dialectic logic. 


son. A social action is an action, in which both relata, agent and patient, are 
persons or one of them is a social system or public good. Agent and patient 
are defined as relata of the action relation. In an action relation “x acts upon 
y” the relatum x is called “the agent” and y “the patient”, if x acts upon y. Both 
entities are said to “interact” in case the patient y reacts back on the agent x 
that initiated the process, i.e., y acts upon x, and y becomes the agent and x 
the patient. In such a case, except for practical purposes, the agent/patient 
distinction disappears. Based on these definitions, Bunge defines several 
other concepts associated with “action” such as “consequence of an action” 
and “reaction”. 

As exemplified by Mario Bunge’s definition of terms such as “action” and 
“interaction”, the meanings of “action” and “interaction” encompass human 
and non-human actions. Human actions and interactions form a special 
case of the broader definition of “action” and “interaction” and, in general, for 
the relata of interactions no distinction is made concerning agenthood and 
patienthood. 


Interactivity and 
Interaction as Symmetrical Relations 


Even if it is not necessary for action and agency to be associated with 
humans, as shown by the definitions given by Mario Bunge, it can be seen 
in sociology and philosophy of technology that these terms are often only 
ascribed to humans and not to machines. For example, in the German syn- 
thetic or pragmatic philosophy of technology developed by Hans Lenk and 
Jürgen Ropohl, humans interacting with machines in order to achieve a goal 
are considered to form an integral system, a socio-technological action unit.'* 
In this view, even though a machine and a human form an integral action 
unit, the machine only concurs to the human action. A machine is not con- 
sidered as an agent or actor, because it lacks intentionality and (human) 
purpose. This restriction implies that the action relation is thought to be 
asymmetric concerning the kinds of relata. Only a special kind of relata can 
be agents. Taking into account particularly recent directions of research 
into human-computer interaction (HCI) and human-robot interaction (HRI), 
concepts or theories that assign activity only to humans and passivity to 
machines seem dubious. 

Bruno Latour’s actor-network theory and socionics,!® an approach to 
sociology which combines computer science and sociology, are in contrast 


200 zZ 


14 The German term is “soziotechnisches System”. 


15 Werner Rammert’s contribution addresses some implications of socionics for 
interaction. 
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to Hans Lenk’s and Jürgen Ropohl’s synthetic and pragmatic philosophy of 
technology.! These theoretical approaches propose considering the action 
relation between humans and machines to be symmetrical, and advocate a 
kind of anthropology, especially a symmetrical anthropology, which views the 
roles of machines in human-machine interaction in general to be equated 
with human roles. 

The insight into the viewing of human-machine systems as integrated 
systems has been made by Arthur W. Burks for computers in connection 
with the most effective use of both humans and computers. He uses the 
term “human-computer combines”!” and points out the importance of the 
social implications of their use: “Electronic computers are the first active or 
“live” mathematical systems. ... The most effective use of computer programs 
is to instruct computers in tasks for which they are superior to humans. 
Computers are being designed and programmed to cooperate with humans 
so that the calculation, storage, and judgment capabilities of the two are 
synthesized. The powers of such human-computer combines will increase at 
an exponential rate as computers become faster, more powerful, and easier 
to use, while at the same time becoming smaller and cheaper.'® The social 
implications of this are very important.” 

So far, our discussion of human-machine interaction has revealed two 
important aspects of the human-machine relationship: 1) In general, there 
is no logical necessity to associate “action”, “interaction”, and “interactiv- 
ity” only with humans, especially not in the case of human-computer and 
human-robot interaction. 2) Furthermore, human-machine systems form an 
integrated system that increases the power of both human and machines. 

Concerning the goals of artificial intelligence, the idea of thinking machines 
and the discussions about human and machine intelligence, Harel makes 
the following distinction: “Perhaps, instead of AI, “artificial intelligence”, the 
emphasis should be on IA, intelligence augmentation, which is the develop- 
ment of computerized tools that enhance human intelligence and improve its 
functioning. Combining the best aspects of human and machine may bring 
about that neither can do in its own.”!® 

Burks describes this situation for scientific research on goal-directed and 
intentional systems for developing robots: “The ways in which models are 
used by goal-directed systems to solve problems and adapt to their envi- 
ronment are currently being modelled by human-computer combines. Since 
16 See Maring 2008, p. 118. 

17 Burks 1999, p. 167 
18 This indicates the relevance of the human-in-the-loop. 


19 Harel 2004, p. 400. Technologically speaking, at present most “interactive” systems 
for social or artistic interactions are reactive systems. See Harel 2004 for more informa- 


tion on reactive systems. 


computer software can be converted into hardware, successful simulations 
of adaptive uses of models could be incorporated into the design of a robot. 
Human intentionality involves the use of model of oneself in relation to other 
and the environment. A problem-solving robot using such a model would con- 
stitute an important step toward a robot with full human powers.”?° 

Science, especially cognitive science and research on human-computer 
and human-robot interaction, uses interactive art as a test bed in order to 
study action, perception, and cognition. This idea is elaborated in the next 
paragraph. 


Interactivity and Cognition: 
Environments, Affordances, and Effectivities 


It is important to note that the internal model of oneself is used in relation 
to others and not only to an environment. Furthermore, it must be pointed 
out that human interaction and communication takes place in a social and 
cultural environment rather than in a biological or physical environment. A 
social or cultural environment differs in many respects from a natural envi- 
ronment. The most important difference seems to be the use of symbolisation 
in social and cultural interaction and communication. 

An agent’s situatedness and “interactions” with an environment are 
highly important for the study of cognitive and perceptual capacities. In the 
context of these studies the concept of “affordances” is essential. The idea 
of “affordances” plays an important role both in the study of human-com- 
puter interaction?! and interaction design and in neuro-robotics and cogni- 
tive and behavioural robotics. Gibson introduced the term “affordance” to the 
psychology of perception, and Norman to human-computer interaction and 
interaction design. For Gibson affordances are action possibilities available 
in an environment, independent of an agent’s ability to perceive these pos- 
sibilities. These are the actual possibilities of the environment. Affordances 
are conceived of as information in the sensory stream concerning opportuni- 


220, zZ 


20 Burks 1999, p. 168 


21 The use of methods from human-computer interaction in non-technological con- 
text in combining HCI research and interactive art is a new emerging field, e.g. Höök/ 
Sengers/Andersson 2003. At first glance there seems to be a difference between actions 
in everyday life, at work and art. It is possible to think of a work of art as consisting 
of interacting objects that humans are passively experiencing. In such a case humans 
are patients, and the artwork is the agent acting upon humans. Another more general 
scenario is that humans play an essential role in interactive art and are acting upon the 
objects and machines: This is the human-in-the-loop. In interactive art the human is 
necessary for interaction. 
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ties for action in and provided by the environment. Norman uses the term 
“affordances” for perceived possibilities, even if they may not actually exist.?? 

“Effectivities” is the concept complementary to “affordances”. They “are 
the range of possible deployments of the organism’s degrees of freedom,” and 
“... the development of novel effectivities creates opportunities for the recogni- 
tion of new affordances, and vice versa.”?? 

In general, affordances provide cues to the operation of objects. They are 
the link between tools or objects and the knowledge of their use, i.e. the 
operational chains. 

Effectivities expand the range of affordances. Gibson’s and Norman’s con- 
cepts of “affordances” neglect affordances in the case of social interaction, 
i.e. where the tools or objects are robots or humans. It might be a good idea 
to expand the ideas of “affordances” and “effectivities” to the study of social 
human-machine interaction, especially human-robot interaction and media 
art. To what extent are the actions of others in social interaction guided by 
affordances and effectivities? What the effectivities and affordances in social 
and emotional interactions with machines are remains an open question for 
research. 

As noted previously, new media art and interaction art are increasingly 
being used as a test bed for scientific research. In interactive and media 
art human-computer combines can be used to enhance artistic productiv- 
ity: artistic human-computer combines may form expression and structures 
that humans or computers alone can’t achieve. At the same time, in applying 
theories and methods provided by the sciences, art explores the capacities 
of humans to sense, perceive, and act in unknown environments: Scientists 
become artists and artists become scientists. Therefore, in using human- 
computer combines and robots in science and art, it seems that the boundary 
between science and art is increasingly being blurred. One claim concern- 
ing this development is that art and science - as in the Renaissance” - are 
beginning to form a new alliance.” The artistic use of current developments 
in robotics,” artificial life,” software algorithms,” and agents” as well as 


22 For a detailed analysis of different theoretical uses of “affordance” in Gibson 1979 
and Norman 1988 see Gaver 1991 and McGrenere/Ho 2000. 


23 Arbib 2006, p. 6 

24 See Douglas Robertson’s analysis for science and everyday life. 

25 Hans Diebner’s contribution elaborates this idea. 

26 On the artistic use of robots see the contributions by Suguru Goto; Gil Weinberg. 
27 See the contributions by Jin Hyun Kim; Christoph Lischka. 

28 See the contributions by Frieder Nake; Julian Rohrhuber. 

29 See the contribution by Georg Trogemann, Stefan Göllner, and Lasse Scherffig. 
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mixed, augmented and virtual reality® makes the relocation of the relation- 
ship between humans and artefacts evident. 

New Media Art is seen in this context as a field in which art, science and 
technology are interwoven. As media artists are not only involved in artistic 
practices, but also in investigations analogous to science,®! (artistic) creativity 
is needed in science to deal with new epistemological problems which come 
to focus through newer technologies.” Technologies are not only a means to 
achieve a goal, but rather a mediator for artistic and scientific experiments, 
which serves as component of efficiency respectively effectiveness.** 


Interactivity and Media Theory 


The transformation of technical tools, which began with the advent of 
digital technologies and led to the so-called New Media, changes our habit- 
ual modes of media use, reshapes our experience mediated by media, and 
opens up the possibilities of new designs of artistic and scientific experi- 
ments. Especially, a Media Theory concerning New Media, and particularly 
New Media Art, needs an alternative view to traditional conceptualisations 
of agency and interactivity, within a conceptual framework in which human- 
machine interaction can be based on an asymmetric relation and a co-active 
taking of effect during this interaction can be seriously investigated. In this 
context, rethinking “interactivity” opens a perspective for media theory from 
the point of view of cultural science and humanities which directs a research 
focus towards different themes related to New Media. 

In Germany, there has been a paradigm shift within the humanities so 
that different conceptions of “Medien” (media) have come to the fore since 
the 1990s as a paradigm in contrast to “Geist” (spirit) and “Kultur” (cul- 
ture). Contrary to media computing or psychology, media theory, oriented 
towards cultural science, surveys the operations of media which form and 
constitute the mediatised. Media act as preconditions for cultural semantics 
and psychological experiences. Research focuses were traditionally directed 
towards technical apparatuses or symbolic means, assuming some kind of 
“pure” meaning or intentional communication. Newer approaches of media 
theory do not assume that there is media-free and pre-medial meaning, infor- 
mation or intention that can be conveyed by media. Rather, media not only 


30 See the contributions by Antonio Camurri, Barbara Mazzarino, and Gualtiero Volpe; 
Monika Fleischmann and Wolfgang Strauss; Ludwig Jager and Jin Hyun Kim; Sybille 
Kramer; Martina Leeker. 


31 See the contributions by Hans Diebner; Julian Rohrhuber. 
32 See the contributions by Hans Diebner; Christoph Lischka. 


33 See the contributions by Werner Rammert; Julian Rohrhuber. 
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act as an indifferent means of conveying the mediatised, but also participate 
in its shaping.** Therefore, “mediality” as the main operation of media which 
refers to the relation of a medium to the mediatised comes to the fore. The 
traditional concept of interactivity is not commensurable with the basic idea 
underlying this paradigm for media research within cultural science. 

The main research interest of media theory related to interactive media 
is the question of how different media formats may have an effect on the 
meaning formation, information or experience generated co-actively by inter- 
actants.°° For instance, how the practices of virtuality offer the possibility of 
interactivity with symbol structures, makes clear the difference of compu- 
ter-based media from literal media.‘ Information technological and artistic 
experiments with HCI and HRI can therefore be investigated in respect of 
medial operations of newer technologies? which can not only serve as an 
analysis of mediality of New Media, but also have an impact on information 
technological research on HCI and HRI and artistic practices. 


Interactivity and Emotion: Relational Artefacts 


So far, interactivity and interaction have been dealt with in connection 
with the logical problem of defining “action” and “interaction” and discus- 
sions in the philosophy and sociology of technology concerning its relational 
property as being either symmetric or asymmetric. Furthermore, the impor- 
tance of considering human-machine combines as integrated systems and, 
especially for human-computer interaction, as augmenting the power of both 
humans and computers for scientific research and artistic projects was men- 
tioned. The convergence of art and science in using interactive art as a test 
bed for testing scientific hypotheses was noted. The importance of distin- 
guishing between a natural and a social or cultural environment was pointed 
out. This distinction is based on symbolisation for communication, informa- 
tion exchange and transmission in cultural or social environments. Media 
theory was introduced as a point of view from the humanities concerned 
with the role of media in symbolisation and meaning formation. However, so 
far, the importance of emotional and social interaction between humans and 
machines has been neglected. 

In the epilogue of the 2004 twentieth-anniversary edition of her famous 
“The Second Self: Computers and the Human Spirit” from 1984 the psycho- 
analyst Sherry Turkle reflects on the present situation concerning the rela- 


34 Tholen 2005, p. 166 

35 See Martina Leeker’s contribution. 

36 See Sybille Kramer’s contribution. 

37 See the contribution by Ludwig Jäger and Jin Hyun Kim. 
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tion between humans and machines. For her the main question is not about 
the real emotional and intellectual capacities of machines, but rather about 
human vulnerability and the human self-image. In analysing the present 
human-machine relationship and the current technological and social devel- 
opments, she coined the term “relational artifacts” for artefacts that “ask their 
users to see them not as tools but as companions, as subjects in their own 
right”! that “... present themselves as sentient and feeling creatures, ready for 
relationship.” For Sherry Turkle “The new questions are not about whether 
relational artifacts will really have intelligence and emotions but about what 
they evoke in their users.” The question concerns “..., what we will be like”! 
and not what computers or robots can do. Furthermore, she points out that 
social and emotional interaction with machine is no longer science fiction, 
but rather social reality, and may affect the way humans think about them- 
selves and their social relations: “The introduction of robotic helpers in nurs- 
ing homes, ..., is now being presented in the United States as potential social 
policy. [...] How will interacting with relational artifacts affect people’s way of 
thinking about what, if anything, makes people special? The sight of children 
and the elderly exchanging tenderness with robotic pets brings science fic- 
tion into everyday life and technophilosophy down to earth. The question is 
not whether children will love their pet robots more than their real life pets or 
even their parents,*? but rather, what will loving come to mean.”** 

The last paragraphs show that, in interacting, human and computers 
form an integrated whole, and it seems appropriate to conceptualise the 
interaction of humans and computers as a symmetric relation. Computers 
are not only conceived of as tools; rather they are best conceptualised as 
“partners”, because computers and robots are increasingly, just like humans, 
acting in social and cultural environments. Symbolisation was identified as 
important in order to distinguish interaction with natural environments from 
interaction in social or cultural environments. As a final step the current 
developments in technology and interaction are put into a historical and evo- 
lutionary perspective of humankind. 


38 Turkle 2005, p. 289 

39 Turkle 2005, p. 288 

40 Turkle 2005, p. 294; Fellous/Arbib 2005 contains further information of the current 
state of the art in research on the human brain, emotions, and robots. Social human- 
machine interaction based on detection and emulation of emotional states is treated e.g. 
in Picard 2002 and Breazeal 2002. Dautenhahn 2002, 2007 and Dautenhahn et al. 2002 
treat social aspects of human-machine interaction in general. 

41 Turkle 2005, p. 294 

42 One may speculate that children will like their robot teachers more than their 


human teachers. For more information on learning, children, and robots, see Druin/ 
Hendler 2000. 


43 Turkle 2005, pp. 295 
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Human-Machine Interaction: A Broader View 


A perspective which is not dealt with in this book but which underlies its 
conception is that tool use, symbolisation, and the evolution of the human 
mind are interwoven.** It is hypothesised that humankind is in a new phase 
of its cultural evolution which started with tool use and language 100 thou- 
sand years BP continued with the invention of script 4000 years BC, and 
printing in 1500 AD and digital technology in the 20th century. 

In Anthropology and prehistory the speciation of humankind has been 
associated with tool use, bipedality, increasing brain size and lateralisation, 
and symbolisation, to mention just a few of the proposed characteristics that 
distinguish humans from monkeys and apes. But now there is increasing evi- 
dence from primatology, anthropology and prehistory that social intelligence, 
interaction, and communication seem to be the causes of the differences that 
distinguish the hominin line from the other hominids. Since the advent of the 
human species, biological evolution has become more and more a cultural 
“evolution” in connection with symbolisation* and the tools for communica- 
tion, information exchange and cultural transmission. The invention of writ- 
ing systems and the printing press are well-known examples. But one has 
to bear in mind that these tools serve to facilitate social purposes such as 
cultural communication, information transmission and exchange. 

The important point to note is that digital information technology operates 
in social realms of interaction, intelligence, and communication. Furthermore, 
it is important to note that it is not the material culture, i.e. the physical 
objects, in itself that is of importance, but rather the procedural knowledge 
associated with their use: the techniques or (procedural) knowledge of their 
use. The ethnologist Edwin Hutchins uses the term “cognitive artifact” as a 
concept which “... points not so much to a category of objects, as to a cat- 
egory of processes that produce cognitive effects by bringing functional skills 
into coordination with various kinds of structures.” Similarly in relation 
to tools and their use the French archaeologist Leroi-Gourhan pointed out 
the relevance of operational chains, i.e. operational sequences of technical 
actions.*” 

This may raise the question: What are the operational chains or cogni- 
tive artefacts which guide social interactions with machines, especially in 


nn 
44 On human evolution, technology, and cognition see e.g. Audouze 1999; Gibson / 
Ingold 1993; Washburn 1960. 


45 Donald 1991, 2001; Lock/Peters 1996 


46 Hutchins 1999, p. 127. There seems to be a close connection to some uses of 
“media” and the idea of mediality in media theory. Unfortunately this relation cannot be 
elaborated here. 


47 Leroi-Gourhan 1964; the French term is “chaînes opératoires”. 
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an artistic context, and how do they change and develop in social human- 
computer and human-robot interaction? In general, how are they identified? 
All these questions are open for further research. 


Conclusion 


Currently we are at a crossroad in the co-evolution of humans and 
machines. We have identified symbolisation or symbolic communication 
and social interaction as the core of this co-evolution. It is claimed that the 
arts, especially New Media Art, in connection with physical computing, social 
robotics, and human-robot interaction, are becoming an extended “labora- 
tory” for scientific research on social interaction and the human mind and its 
underlying psychological and neuronal mechanisms, as well as the cultural 
origins of higher cognitive functions. At the same time they are exploring 
new effectivities and affordances in the social or cultural art environments. 
Furthermore, interactivity in human-machine interaction is no longer merely 
a technological issue, or one only for scientists and engineers. Neither is inter- 
action a topic only for psychologists and sociologists. The effects and conse- 
quences of human-computer and human-robot interaction are becoming an 
issue concerning all aspects of social human life and existence. Especially 
concerning the design of robots and human-robot interaction, ethical top- 
ics must be urgently addressed. Because of its impact on social commu- 
nication and structure and the importance for the human self-image, new 
conceptualisations for describing, analysing, and theory-forming, as well as 
empirical research methods, are urgently needed to study this development. 
Media art in connection with cognitive and media science, human-computer 
and human-robot interaction is one of the best ways to cope with this need. 
It is not false to predict that in the near future the importance of its social, 
educational, political, philosophical, and theological implication will become 
tremendous. For example, Sherry Turkle observes: “Both psychoanalysis and 
computation challenge common sense understandings of action and respon- 
sibility because they get people thinking of a ‘decentered’ self - a self that is 
not a unitary, intentional agent.”** The co-evolution of humans and machines 
will have essential effects on the human mind. Therefore, it is important to 


“ 


bear in mind Merlin Donald’s assertion: “... the role of the individual mind 
is changing, not in trivial ways but in its essence. And these changes need 


watching.”*? 
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48 Turkle 2005, p. 356 
49 Donald 1991, p. 360 
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This book gives a bird’s-eye view on the current situation and some hints 
on what to look for in order to watch.°° 
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1. Dematerialisation? 


The notion that the use of new media effects a dematerialisation con- 
stitutes a kind of lowest common denominator for the various designs of 
knowledge societies’ perspectives: Norbert Wiener’s distinction of information 
from matter and energy turned information into a quantifiable ‘universal coin’ 
that permitted all phenomena to be commensurable and transferable under 
the aspect of their telecommunicational coding, regardless of their spatio- 
temporal situatedness and their respective meaning. Marshall McLuhan’s 
‘global village’ proceeds from the assumption of an electronic shrinkage of 
spatial distance which enables the realisation of a form of telepresence that 
supersedes the principle of locality as a sine qua non of interactive commu- 
nication.! Hans Moravec’s utopia of ‘mind children’ projects a transplantation 
of our mental capability into machines, so that intelligence and information 
become independent of biological embodiment, and that the body, just as the 
brain, may degenerate into the dross of an immortalised mind.? Then Jean- 
Francois Lyotard’s Parisian exhibition ‘Les Immatériaux’ (1985) staged this 
tendency towards immaterialisation as a topos in the borderland of science 
and art, significant for out time.? The assumption that the informatisation of 
the lived-in world leads to a dematerialisation is accompanied by a rhetoric of 
disappearance: the unity-endowing narrations drop away, the senses dwin- 
dle, the signs lose their referents, reality evaporates into hyperreality. Our 
culture’s unconcealed interest in soma - be it in terms of athletical stylisation 
of the body or as an endorphin-raising, extreme bodily experience — can then 
be interpreted as a compensation and counter movement to this apparent 
disembodiment which is supposedly brought about by new media: a fascina- 
tion both confirmed and approved of simultaneously insofar as these bodily 
practices are only possible in spaces that are three-dimensional and hazard- 
ous, thus real and not virtual. 

The following considerations should be taken as a critical comment of 
the presumption that new media brings about a dematerialisation and dis- 
embodiment of our civilisation. The idea of the disappearance of the body 
under conditions of virtualisation falls short. It is not the dissolution of the 
body taking place under these conditions, but rather its splitting into both 
a human body and a data body, its doubling into a physical and a semiotic 
body. Occurrences in virtual reality - we hereafter make use of the expression 
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‘virtual reality’ as a technical term? — are not simply due to a disembodiment, 
but to the sublime transformation of the body into flesh body and sign body. 
How can we trace this transformation? Spatiality and the principle of local- 
ity play a role. Descartes defined matter by spatial extension and therewith 
specified as the concept of body that which occupies a well-defined place 
in space. If now the conception of the body is subject to change, then this 
change must - in some way — have to do with a transformation of space. The 
metamorphosis of ‘corporeality’ under the condition of virtualisation relates 
to a metamorphosis of ‘spatiality’ itself. 


2. Vertigo in virtual space? 


There is a remarkable phenomenon which Jay David Bolter calls atten- 
tion to: there is barely one sensation which more sustainably and radically 
asserts our having a body and its actual determination by its position in 
space than the phenomenon of vertigo. People who suffer from acrophobia 
are hardly capable of stepping on high balconies, climbing towers, or taking 
transparent lifts. Being exposed to such circumstances causes somatic afflic- 
tions which can assume the characteristics of pathological anxiety reactions: 
heart complaint, sweating hands, feeling of oppression, paralysis.° Yet fear of 
heights is only perceived when one’s body is actually placed in an elevated 
position. If virtual realities depend upon disembodiment, acrophobics who 
step on a virtual suspension bridge or enter virtual lifts via immersion tech- 
niques should not suffer from any physical discomfort. However, that is not 
the case. At Georgia Institute of Technology, three computer-generated virtual 
spaces were designed: a balcony situated high-up and providing a downward 
view, an open lift, and building-to-building suspension bridges. Acrophobia 
patients who entered these virtual installations did not only exhibit each 
somatic symptom but were even cured with the aid of these virtual spaces. 
Thus, the mere simulated presence of the body in the data world is per- 
ceived as a real presence and triggers physical, anxious reactions in one’s 
own body. 

How can that be? Bolter, with good reason, suspects this is not due to 
realistic depiction, for the phobia producing virtual realities are strongly sim- 
plified and at best coarsely operating with photo realistic, illusionary tech- 
niques. Rather, what counts is the fact that the setting ‘stepped into’ reacts to 


4 By the term “virtual reality” we refer to technologies which allow for the integration 
of a user into computer generated environments, therewith enabling him to interact with 
the data universe. On this term, see Buhl 1996, p. 53. 


5 Bolter 1996 
6 Hodges et al. 1995 
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the user’s respective body movements. Obviously, an interaction takes place 
between the user’s physical body, fitted with a data suit and a head-mounted 
display (HMD), and his virtual body, situated on the suspension bridge. But 
this virtual body is only present in terms of the respective viewpoint from 
which the virtual environment presents itself to the user’s eyes via HMD. 
Hence, in this case, the interaction is only present and embodied in terms of 
the perspectives which are constantly changed in respect to the movements 
of the user’s physical body. Nevertheless, whilst the actual body only seems 
to be high up, the semiotic body - taken to be a point of view from a certain 
altitude - is actually present on the suspension bridge. 

Let’s have a closer look at this relation between virtuality, illusory placing 
and interaction. 


3. What does ‘virtuality’ mean? 


Elena Esposito originated the idea of associating ‘virtuality’ with illu- 
sory placing thereby introducing the well acquainted phenomenon of mirror 
images to explain ‘virtual’.’ ‘Virtual’ is a term in optics which refers to images 
not based on light rays spreading linearly. Reflections, for instance, are vir- 
tual, insofar as they convey the impression of the mirrored objects as if they 
were actually located behind the mirror surface. Thus virtuality not only pro- 
duces illusory objects, but provides real objects with illusory placings. In this 
way, seeing things from front and behind simultaneously, or seeing oneself 
with the eyes of the others, becomes possible. Even so we have no chance 
of stepping into the mirror image. We have to change the world in front of 
the mirror for the world in the mirror to be changed. Indeed, reflections are 
images, but not signs of the reflected objects.® By extrapolating the mirror 
metaphor we can make a first attempt to grasp what ‘computer generated vir- 
tualisation’ means. Imagine that there is a technology (a) allowing us to step 
into the mirror world and to interact with the mirrored objects, whilst (b) what 
is mirrored is not simply things, but signs resp. symbol worlds. Computer 
generated virtual realities are immersive reflections of symbolic universes, in 
which a user can interact with symbol structures. 

However, there is a bottleneck that must be passed through from the real 
to the virtual and vice versa, and this is only achievable by semiotisation. 
For the user this means that an immersion in the virtual world can only 
succeed provided a mapping of his corporeal body to a data body, which acts 
as an arbitrary symbolic (re-)construction of the viewpoint and movement of 
the physical body, is carried out. This can be accomplished by information 
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technologically upgrading the body with a data suit, helmet and gloves, or 
more subtly, via scanning or detecting the body by means of video cameras. 
What it comes down to is that positions and movements of body parts are 
recorded, digitised and made present by a “sign body” in virtual space as a 
representation of the physical body. This presence of the “sign body” can be 
implicit in terms of the viewpoint which determines the respective perspective 
under which a scenery appears as a function of the user’s ocular movements. 
Or this presence can be explicit in terms of an arrow, a stylised hand, or any 
fictional figure which becomes part of the virtual scene. In any case the user’s 
body exists as twofold, the physical body and the semiotic body. This dupli- 
cation only succeeds since “flesh body” and “sign body” are connected by a 
bilaterally permeable, electronic umbilical cord. 

But what gets interchanged by means of this “umbilical cord”? All virtual 
reality input technologies come down to the transmission of ocular, manual, 
facial and bodily movements. Usually, by “body” we mean something which 
has a specific position in space and time. Regarding bodies which change 
position we speak of movement. Movement, like the body itself, is determined 
by the principle of locality, insofar as movement invariably relates to a change 
in position of a body. Therefore, the user’s presence in three-dimensional 
space and his performing actual movements — however limited the latter may 
be - is the condition for his semiotic body to become active in cyberspace at 
all. Thanks to the computer’s computational power, the bodily movements 
are transferred to the data body so fast as to be unnoticeable to human 
perception, any sense of a transmission vanishes. Thus the user’s semiotic 
body experientially does not figure as a counterpart to his physical body. 
The intimate spatiality, in which bodies relate to one another positionally by 
“here” and “there”, does not hold anymore for the relation between physical 
and semiotic body. 

It is exactly this exemption of the principle of locality concerning the data 
body which is the artistic message of the interactive computer installation The 
Trace, Remote Insinuated Presence, presented in 1995 at the Contemporary 
Art Fair in Madrid.? Within this work, which is described by the artists as 
“tele-embodiment”!°, participants who are located at separate places can, at 
their respective place, encounter the other’s semiotic double. The physical 
body of one participant and the semiotic body of a second can approximate or 
evade one another, and eventually occupy the same place by the coincidental 
positions of the real person in three-dimensional space and the virtual per- 
son in telematic space. That way, real body and data body can “merge” resp. 


9 This installation was presented by media artists Rafael Lozano-Hemmer and Will 
Bauer at Contemporary Art Fair in Madrid: Lozano-Hemmer 1996. 


10 “Tele-embodiment is the technically supported act of being in spatial and temporal 
coincidence with other humans.” Lozano-Hemmer 1996, p. 142. 
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incorporate: they coincide. The principle (valid for physical reality) that two 
bodies cannot be in the same place at the same time does not apply anymore: 
The “flesh body” of one person, and the “sign body” of another are in the same 
“position”. The imperative of distance does not hold. ‘Illusory placing’, which 
we introduced as a defining criterion of the virtual, consists of just this trick 
of zero distance. 

But is that which is placeable in an illusory manner still a body at all? 
And is where it is placed, still a space at all? Differently put, if we continue 
calling the data body “body” and calling virtual spaces “spaces”, how does 
the concept of body and the concept of space have to change for such des- 
ignations not to become meaningless? Answering these questions remains a 
research task, but at least some directions emerge. For one thing, the con- 
ceptual transformations have to do with the altered relation between space 
and time — space here taken as three-dimensional, physical space - and for 
another thing, they have to do with the relation between space and person — 
space here taken as social space. 


4. Space and time: The implementation 
of time into sign configurations 


What does the proficiency of computer generated virtuality consist of? 
The buzzword “interactivity” can show us a way. What “interaction” means 
is familiar from verbal conversation: ‘Ego’ and Alter’ reciprocally refer to one 
another in what they say and how they say it. And we are just as acquainted 
with the intervention of writing to split up communication and interaction: 
wherever is written, the readers are absent, wherever is read, the author is. 
Indeed the symbolic register of literality (we can also say, cultural techniques) 
rests upon our ability to produce, look at, convert, interpret and erase signs. 
And yet they still elude interaction. We know about interacting with persons, 
with animals, too, and even with things on a limited scale, but an interactiv- 
ity involving symbols is unknown to us. And that does not only bear on our 
dealing with texts, but likewise includes our relation to images." 

Is it by chance that spatiality is the fundamental medium of this symbolic 
register of literacy? A complex link between the culture of literacy and the 


11 However, there is a misunderstanding to pre-empt: the dispension of interaction is 
not just lack and loss, but a culture conveying achievement which opens up new scopes 
to our cognitive and aesthetic world-relatedness. In conversational interaction among 
physically present persons, characterised and also blurred by asymmetries of charisma 
and power, it is just its interruption which in the form of solitary writing and reading 
profoundly promoted an individuality in the sense of the formation of an idiosyncratic 
perspective, critical faculties towards the truth claims of texts, as well as an insight into 
the difference and plurality of interpretational possibilities. 


medial order of spatiality is touched upon here. There are several phenomena 
which testify to such a connection: 


(1) For one thing, there is the privileging of eyesight, dating back to 
Greek Antiquity.!? This, however, is a sense by which the manifold can 
be surveyed side by side, i.e. simultaneously, and whose capability 
furthermore increases, not diminishes, with the distance to the viewed 
object. 

(2) For another thing, there is the — mostly remaining implicit — identi- 
fication of language with script (Schrift): nearly all theories of language 
draw their idea of language from the model of script, which, however, is 
a language modality which makes use of the potential for representa- 
tion of spatio-visual configurations by which language advances to an 
image, namely to a script figure (Schriftbild). 

(3) And lastly, there is the fact that we even represent time itself only by 
means of visual schemes, be it a point of time, an arrow of time, or a 
space of time: whenever we indicate temporality, time is spatialised. 


The use of computers presents a kind of cultural technique inasmuch as 
the cultural techniques of literacy such as reading, writing and arithmetic 
rest upon the exclusion of interaction with symbols, whereas that of compu- 
ter utilisation, however, rests upon the inclusion of interacting with symbols. 
This ability to interact with symbol structures — and that is the supposition 
that it comes down to - becomes possible only by the implementation of time 
into symbolic configurations. Well, how is that to be understood? 

Virtually all metaphors used to describe the forms and practices that 
have arisen in dealing with information called on conceptions of space (be it 
the “data universe”, the “net”, or the “desktop”) as an organisational principle 
of a computer’s user interface. Yet the assumption of this preference for the 
semantic field of spatiality does not account for the future impulses by which 
computers will shape our access to and handling of information - impulses 
which consist in the implementation of temporality into forms of symbols we 
traditionally organised as spatial relations. Writing, text and the book: of these 
media it is a characteristic — unlike the fluid word, barely uttered, already 
fading away - to display a time-resistant stability. Although each reading 
process creates a unique text for the reader and, during repeated reading a 
varied one as well, the texture conceived as a legible structure of significants 
remains stable. Therefore, if the transition”? from orality to literacy in medial 
Cu 


13 The word “transition” here does not mean a substitutional relation of orality by lit- 
eracy, of which there is none in this naive form, but is meant in the sense of a systematic 
distinguishability. 
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respects can be described as a spatialisation, then the transition from literacy 
to telematics lies in a temporalisation. 

David Gelernter assumes that we are standing at the verge of a period in 
which computers mainly “deal with palpable time - with visualised, concrete 
time.”!* There is something static and immanent to space, yet time flows (by). 
The fluent current, i.e. no longer the cabinet and the document kept therein, 
becomes the essential organisational principle of information. If temporal- 
ity becomes inherent in data structures - this being Gelernter’s speculation 
- information which is accessible and processable via computers does not 
remain in the shape of “documents” but takes on that of “cyber bodies”. It is 
then organised like a “life stream”. Approaches to a non-Cartesian conception 
of the body become visible here: a body is taken as an entity in flow which is 
not defined anymore by its position in space, but by its changing in time. 

The basic idea of this speculation, that the implementation of time into 
data structures would make a significant difference between the “register of 
textuality” and the “register of digitality”, already becomes unspeculatively 
evident with, for example, image-generating techniques in computer gener- 
ated numerical simulations, which fulfill visualisational functions in the sci- 
ences. Gabriele Gramelsberger showed’ the basis of this kind of simulation 
to be a novel form and function of script, which she designated “digitised 
script”. By means of digitised script it becomes possible to not just describe, 
as with phonetic script, nor to just calculate, as with formal, operational 
script, but to dynamise system flows in order to render temporal processes 
functionally representable, and computer animated images analogously pre- 
sentable as well. This dynamisation of data structures in numerical simula- 
tion rests upon the fact that “points in time become real numbers which are 
defined by infinitesimal processes of approximation”:'° thus, time becomes 
integrated in mathematical - and with this also symbolical — structures. 

This simulation technique also allows for user movements — whether that 
involves eye movement, facial expression, gesticulation, or whole body move- 
ment - to be captured in a way that movement of the physical body becomes 
simulateable in virtual space by the data body. This data body is “corporeal” 
not only because it bears a relation of mapping to the user’s body, but also 
because time is implemented into the symbolic structures of this representa- 
tion as well as into the symbolic environment itself. The symbols have become 
dynamical, they are incarnations of time, and it is just that which enables 
novel interactivity with them. 
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14 Gelernter 2000, p. 59 (translated quoting) 
15 Gramelsberger 2002, p. 75 
16 Gramelsberger 2002, p. 71 (translated quoting) 
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5. Space and body: the constitution of corporeity 
by central perspective and virtual reality 


Through his distinction of “res extensa” and “res cogitans”, Descartes did 
not only determine the “body” by its difference to mind, but at the same 
time he grasped the notion of matter as spatial extension, i.e. identifying the 
physical body with a geometric body. 

Now if the data body in the web owes its corporeity to the implementa- 
tion of temporality and motion, does that make virtual reality a phenomenon 
which undermines the Cartesian body conception? One might think so. Yet 
the situation is more complicated with regard to current techniques of virtual 
reality, since the virtual body is dynamisable only insofar as the real body, 
placed in three-dimensional space, gets reduced to a “Cartesian body par 
excellence”. How is that to be understood? 

At this point, a revealing analogy between virtual reality and the technique 
of central perspective lends a helping hand. With his conception of matter, 
Descartes philosophically realises that which central perspective effectuated 
with regard to the early modern body image. Usually, depiction employing 
central perspective appears as a “natural” representation resting upon an 
imitation of the laws of the visual process by the rules of artistic creation. 
But space in central perspective is an infinite, continuous, homogenous, 
and thus mathematical space which to no degree coincides with the psycho- 
physical space of human corporeity, to which top and bottom, left and right, 
front and back are precisely not homogenous. Linear perspective creates a 
visual syntax, in the medium of which the body becomes determined and 
visualised in a new way: bodies - as well as their spacings, by the way - rank 
as incarnations of geometric proportions by which they are homogenised and 
rendered comparable. The body is embodied mathematics. Accordingly, the 
representation of the body in a picture exercising central perspective is not 
founded anymore on the significance and relevance of the depicted persons, 
but exclusively depends on their position in space put into an arithmetical 
relation to the immobile eye of an external observer. 

Maybe the ‘family likeness’ between central perspective and virtual reality 
can now become clear. Regardless of the crucial difference that tableaux rest 
upon separation whereas virtual realities rely on the concurrence of represen- 
tation and interaction, there is a similarity. It consists in the fact that what is 
considered a body does not only attain representation, but at the same time 
is constituted by techniques of representation. The doubling into the physi- 
cal and the semiotic body as a condition for interacting with virtual realities 
means that the body of the user has to be transformed into a purely physical 
body. The coordinates of the moving physical body are transferred to its semi- 
otic double in electronic space. The user “counts” as a mere body-in-motion. 
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Like central perspective transforming the resting body into something funda- 
mentally characterised by its spatial position, virtual reality transforms the 
mobile body into something ascribed by its digitisable motion grid. So, would 
it not be true to say that the clue to virtualisation also lies in disembodiment 
and depersonalisation, and not solely in the dissociation of physical body 
and data body? 


6. On the transformation 
of social space in telematic communication 


6.1 Script as a medium of synchronicity: 
the “pseudonym-isation” of communication 


Now we have to focus our interest on another phenomenon concerning 
the use of computers. This time it is neither about virtual realities, in the 
sense of immersive data spaces, nor their relation to the user’s three-dimen- 
sional, physical space. What it is about is the telematic, synchronous com- 
munication between spatially separated persons, taken as the genesis of a 
social space of virtual communities. 

Social spaces come into existence through communication; what they are 
made of are all conditions whose ‘being’ rests upon ‘being acknowledged’. 
From the perspective of acknowledgement relations, humans in social spaces 
do not just rate as a biological species, nor just as corporeal beings, but 
rather as persons. If computer mediated telecommunication leads into a vir- 
tualisation of social spaces, what does that mean, then, for the status of the 
communicators’ personhood? 

In order to find an answer to that, we have to look at which shape the 
semiotisation takes, i.e. the inevitable passing through the bottleneck when 
entering into online communication. 

First, script becomes the medium of synchronicity.!’ Usually, script dis- 
rupts communication: writing and reading disengage not only from the imme- 
diateness, spontaneity, irreversible fluidity of actual communication, but also 
from the communicators’ entanglement. Between the production and recep- 
tion of texts a time interval intervenes. And that even holds for email or par- 
taking in Usenet'*, both of which are kinds of asynchronous communication 
where the participants are not simultaneously involved in communication. 
This changes with telematic communication in which script — or more pre- 
cisely the use of keyboard and display — obtains a new purpose, insofar as it 
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17 As to the altered role of writing, see Bolter 1997; Sandbothe 1997. 


18 Usenet taken as an information exchange, where topic-specific articles are inter- 


changed in newsgroups. 


enables the simultaneous exchange between spatially absent persons. With 
this, script enters into functions which, in face-to-face conversation, voice- 
bound speech does fulfil. Yet unlike the indexically meaningful voice, which 
in speaking always asserts the trace of the body, the script figure (Schriftbild) 
entered via the keyboard draws on the potential of anonymisation that is 
linked to script as an option. Due to the relatedness of online communication 
to script, those partaking in it appear pseudonymously. 

We can distinguish two facets of this “pseudonym-isation”: (1) an arbitra- 
risation of the name, and (2) a staging of personal identity. As to (1): no name — 
no account! Having an addressable name is required for participating in web 
interaction. In ordinary life, our proper name is an attribute of our personal- 
ity. It renders ourselves identifiable and gives us a singularity — before any 
biological, psychic or social development of individuality. The proper name 
assigns us a well-defined place in social space and social time. We do not 
create our proper name, but we receive it. And it endows personal identity 
just by virtue of that dimension in which it is not left to our power of disposi- 
tion. This is quite different for the name under which we act in the web: it is 
chosen by ourselves, a product of a self-determined staging. This name pro- 
vides an attributability which can permit remaining anonymous to the person 
connected to it. This name is a depersonalising one. As to (2): the choice of 
a name is complemented by an artificial identity. By the @describe me as 
command we project and produce an identity staged according to outer and 
inner attributes. The arbitrary character of these “self”-descriptions comes 
glaringly to light with the @gender command. Admittedly the web identity 
calls for a specification of gender all along, but which “gender-flag” is set is 
left to the participants. 


6.2 The division of person and persona 


So we can see from the start, the mode of writing in online communica- 
tion facilitates the operation of what are no longer persons but rather arbi- 
trary descriptions of persons, i.e. staged identities behind which all the same 
programmes can be concealed. We shall label this personification of arbi- 
trary descriptions, ‘persona’. ‘Persona’ is derivative of ‘per-sonare’, “to sound 
throug 


speaks his role. So, does the splitting into a real person and a virtual per- 
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and originally means the mask through which the ancient actor 
sona imply a theatricalisation of computer generated communication? Does 


telematic communication become a procedure related to proceedings on a 
stage? 
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Let us start from one popular case that has been frequently discussed in 
the debate on the usage of computers.!? Under the name of legba’ a woman 
from Seattle acts in a MUD” on the Internet. A participant going by the alias 
of ‘Mr. Bungle’ succeeds in cutting the real woman’s control over her alias 
existence by use of a programming trick to then involve her in a violent por- 
nographic interaction. For the public, reading along interestedly, the impres- 
sion is conveyed that ‘legba’ does not only consent to her simulated rape 
but actively partakes in it. After another persona acting under the alias of 
‘starsinger’ is driven to simulated self-mutilation by ‘Mr. Bungle’ using the 
same hack, a ‘wizard’ who has insight into the ‘lower’ code interrupts the 
affair and throws ‘Mr. Bungle’ out of the virtual scene. Then, via a mailing list 
the woman from Seattle makes the events known to the web public; further, 
the virtual community builds an ad-hoc government, which ostracises ‘Mr. 
Bungle’, whose personal identity of course remains unknown. 

So what is the moral of the story? All acting in the web is acting by 
signs and thereby draws on the difference between ‘word’ and ‘thing’. So the 
obnoxiousness of this episode does not lie in the description of rape - for the 
description of rape is not rape, at best a piece of pornography originated with 
this. The obnoxiousness lies in the violently interrupted influence of the real 
person over her virtual persona. 

Yet if that is the case, a crucial difference shows up between a virtual 
communication environment and theatrical events on the stage. An actor is 
indeed responsible for good or bad acting, however, not for whether he plays 
the part of a good or bad character. But even so, a real participant in online 
communication stays in connection with his artificial identity in such a way 
that his responsibility for what he does in communicating, even in the web, 
is not suspended in principle. Nevertheless, the constitutional “pseudonym- 
ity” creates a particular state of affairs that results in the development of 
normative practices in virtual communities which exclusively refer to the web 
personae and by no means - see the treatment of ‘Mr. Bungle’ - to the real 
persons they are associated with. 


6.3 On universal pragmatic theory of communication 


Let us take a step towards contemporary speech act and communication 
theory. Universal pragmatic theory of communication as elaborated by Karl 
Otto Apel and Jürgen Habermas typically links John Langshaw Austin’s idea 
to grasp speaking as a kind of doing, with the idea that a speaker can count 
as a personification of claims to be argued formal-rationally, i.e. a speaking 
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19 Dibbel 1994; Turkle 1995, pp. 250-254; Sandbothe 1998 
20 It is the LamdaMOO created by Pavel Curtis (Xerox PARC). 
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situation being of sublimely juridical resp. contractual structure. Basically, 
this subliminal juridification of communication in universal pragmatic com- 
munication theory is about explaining the detachment of the factual and 
the relational aspect in speech by the fact that in speaking we raise validity 
claims of truth, truthfulness and correctness, which the addressed can then 
— with good reason — accept or reject. The power of the better argument is the 
sole criterion for acceptance or rejection. And it is language itself which - by 
distinguishing ‘communication’ from ‘discourse’, by switching between these 
two levels, and by enabling an interruption of communication and entering 
into discourse — opens the possibility of rationally negotiating and deciding 
the validity claim at issue whenever disagreement arises in the spontaneous 
flow of communication. Discourse becomes the place where - after Habermas 
— abstaining from all corporeal, psychological or social differences, we are 
considered the mere personification of the uniformly distributed ability to 
defend or overthrow validity claims by arguing.” Speaking under the ideal- 
ised conditions of discourse becomes the possibility to be right. This is what is 
meant by the latently juridical structure of communication theory. 


6.4 Personae as participants in virtual communities 


Let us now come back to the question of the “de-personalisation” of web 
communication, by reinterpreting the separation of person and persona from 
the viewpoint of communication theory. 

If the performative dimension of speech is grounded in an inter-subjective 
structure of approval by virtue of which participants in communication are 
personifications of rationally negotiable validity claims, then in anonymised 
web communication this performative dimension is effectively dispensed: in 
telematic communication, it is only spoken and not acted. This is not surpris- 
ing, considering that the semiotisation constitutes the condition of the pos- 
sibility of web presence resp. data existence. From this viewpoint, telematic 
interaction between personae indeed belongs to the type of communication 
which the originators of speech act theory excluded from their reflection as 
a form of ‘parasitic communication’, since here — as in speaking on stage 
— the very propositional-performative double structure of communication is 
suspended. 

But at the same time, speech act and communication theory make use 
of idealisation assumptions which seem to be fulfilled exactly in virtual, not 
in actual life-world communication. Included in this, for example, is that in 
speaking we have to act as beings that are bared of all corporeal, gender-, 
social and geographic differences and who merely have to behave as par- 


A 


21 Habermas 1984, p. 353 
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ticipants provided with equal chances in communication. Where, if notina 
disembodied, “pseudonym-ised”, computer mediated interaction, is such a 
condition given??? 

Regarding the form of communication emerging under such conditions, 
two things attract attention: (a) whereas in the theory of universal pragma- 
tism, “to communicate” and “to argue” concur - at least in discourse — this 
is not the case in web communication. Communicating here is not arguing, 
but is a kind of reference to other personae, by responding to others and 
therefore becoming a participant. The maxim of web communication lies not 
in argumentative rationality, rather in interactive connectability. (b) Accepting 
‘the other as participant in interactive events’ constitutes a kind of acknowl- 
edgment relation that entails the personae to actually form respective issue- 
oriented virtual communities, which then develop their own kinds of rules. 
If, for instance, in order to attract attention a novice within a certain MUD is 
overly hasty with a ‘whuggle’ i.e. a virtual hug, this violates the web-etiquette 
and will lead to the novice being overlooked rather than integrated into the 
chat. 

When relatability and ‘being present’ become the point of reference for 
communication, it has consequences for the modality of community-fostering 
possibilities of regulation and sanctioning. If - as Mike Sandbothe rightly 
emphasises” - it is correct that the internet does not constitute an anar- 
chic space, but rather that proper methods of sanctioning misconduct have 
evolved, then the specificity of these methods consists just in influencing 
this very relatability and ‘being present’ and, if need be, in prohibiting it. The 
scale goes from being ignored by the participants to actual expulsion by a 
‘wizard’. At all times, however, it is a question of a persona being accepted as 
a playmate or not. Communicative interaction in virtual communities has the 
status of game moves.?* 

The question arises whether this phenomenon of our communication 
(aiming as it does, less at the defence of validity claims than at the avoid- 
ance of discontinuation of interaction), does not apply to any communica- 
tion. Moreover, does the divergence of person and persona indicate that in our 
‘ordinary lives’ too, our personality emerges from the sublime interplay of the 
person individuated by its proper name and the persona staged by us? 


A 
22 Unquestionably, other criteria of differentiation in terms of technical expertise 
arise. 


23 Sandbothe 1998, p. 314 


24 For an interpretation of virtuality in context of the conception of “game” see 


Adamowsky 2000. 


7. Conclusion 


The starting point of our reflections was the question whether virtuali- 
sation leads to dematerialisation, disembodiment, and de-personalisation, a 
point about which we were decidedly sceptical. Our argumentation comprised 
of two steps: (1) virtualisation as a possibility of interactivity with symbol struc- 
tures. The practices of literacy familiar to us rest upon our ability to create, 
transform, store, erase, but above all, interpret signs. Yet direct interaction 
with symbolic worlds is excluded by these same practices. Facilitating inter- 
activity with semiotic structures is nevertheless possible through computer- 
mediated virtualisation. And (2) semiotisation of the user as a precondition of 
acting in cyberspace. The user, however, can enter into a synchronous reci- 
procity with signs if and only if he himself is subject to a semiotic metamor- 
phosis. What this semiotic transformation amounts to has to be answered 
differently for the case of virtual reality on the one hand, and that of telematic 
communication on the other. (a) To plunge into virtual realities, the user has 
to split up into a physical body that ‘counts’ as an embodiment of a comput- 
able motion grid, and a data body which, in virtual reality, acts as a symbol 
structure. The movement of the physical body becomes the condition of the 
possibility of activating the data body. ‘Flesh body’ and ‘sign body’ are dis- 
tinguishable but correlate and, by that interplay, introduce the question for 
the modification of our conception of the body. (b) To partake in telematic 
communication, the user has to assume a self-staged identity and split up 
into a real person and a virtual persona. Insofar as the personae depend on 
acknowledgment as participants in online communication, virtual communi- 
ties develop their own standards and practices of sanctioning, which however 
assume the character of game moves, as the exclusion of a persona from the 
communicational events constitutes the most radical kind of sanctioning. 


An edited re-working from Raum — Wissen — Macht, eds. Rudolf Maresch and 
Niels Werber, Frankfurt: Suhrkamp 2002, pp. 49-69 


Translated by Jochen Arne Otto 
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Mediality of artistic computer-human interfaces 


In recent decades, computer-human interfaces (CHIs) have increasingly 
served as a technological condition for New Media Art and at the same time as 
an artistic subject. The design of experimental CHIs, with help of which the 
observer /performer can explore a computer generated artistic environment, 
belongs to an essential part of New Media Art. An interface in the context 
of human-computer interaction (HCI) is defined as a part of the machine 
through which it ‘communicates’ with its environment. An interface medi- 
ates acts of sensory and motor processes of interacting entities. Input inter- 
faces such as keyboards or diverse sensors make external symbolic activities 
accessible to a machine. The processes of machine observation in the form 
of ‘seeing’, ‘listening’ etc. focus on certain modalities of input data streams. 
A digital computer, which is capable of transforming one modality into any 
other, executes an action via output interfaces such as display monitors, 
loudspeakers, or other actuators in accordance with machine observation 
and interpretation of activities mediated by the input interfaces. Interface 
design does not only include hardware design, but also software-technologi- 
cal strategies of mapping from input data into output data. Mapping strate- 
gies are concerned with the question of how sensory and motor processes are 
related to each other by an arbitrary organisation of intermedial translation 
provided by digital technologies. Hence generally CHIs serve as media, not 
only in the sense of technical apparatuses but also in terms of performing 
intermedial translations which act as a condition for the emergence of mean- 
ing and/or experience. In New Media Art, CHIs can be seen in particular as 
media for an artistic experience. In other words, to design an artistic CHI is 
not only an information technological task, but also demands artistic and 
theoretical strategies of mediation, interfacing human and machine percep- 
tion/action, which forms an artistic experience. 

Starting from our thesis that an interface technology is a technology of 
mediation, we will investigate the mediality of artistic CHIs. In using the term 
“mediality” we deal with the question of how CHIs mediate ‘meaning’ and, in 
this way, shape an artistic experience while transferring signals produced by 
human beings and computer systems between real-worlds and computer- 
generated worlds. How then can a CHI act as a medium which is character- 
ised in media theory as a blind spot:' A medium fades into the background 
despite its material presence (e.g. the form of technical apparatus) so that 
the mediatised comes to the foreground, not the medium itself. Taking this 
into account, it is not surprising that “transparency” of a CHI has recently 
been a hot topic of interface design, even from the engineering point of view. 
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1 See for instance Kramer 1998, pp. 73-75. 


But some of the more recent arguments discussed in this context need to be 
reconsidered and called into question. The issue of “transparency”, in the 
design of many of the new interfaces used for interactive music performances 
now take into account ideas about ‘intent’ which is virtually postulated as 
‘pre-existing’ and might be ex-pressed (in terms of externalisation) by motor 
action in dealing with a (transparent) interface.” In our view, dealing with 
‘pre-existent intent’ is far from unproblematic. The core idea underlying the 
design of a transparent (musical) interface, which is however not related to 
intentionality, can be recognised in this early example posed by the composer 
and researcher of computer music, F. Richard Moore in 1988, on how to solve 
the distance between the input and output of a device mapping. He discusses 


this question, introducing the term “control intimacy”: 


Control intimacy determines the match between the variety of musically 
desirable sounds produced and the psychophysiological capabilities of a 
practiced performer. It is based on the performer’s subjective impression on 
the feedback control lag between the moment a sound is heard, a change is 
made by the performer, and the time when the effect of that control change 
is heard.’ 


The recent discussions on “transparency”, however, not only “provides 
an indication for the psychophysiological distance, [...], between the input 
and output of a device mapping”, but also implies “the distance between the 
intent (or perceived intent, in the case of the audience) of the artist to produce 
some output, and fulfillment of that intent through some control action”:* 
the discussions focus on the design of a transparent interface, which offers a 
user practically no distance between intent and action. In this mode of a so- 
called transparent mapping, transparency is often considered an important 
property of an ‘expressive interface’, which is intended to allow a user/per- 
former to mediate her or his (inner) artistic expressiveness. But this concept 
of transparency seems to deviate from the “control intimacy” developed by 
Moore. The latter is based on the notion that “the performer must receive both 
aural and tactile feedback from a musical instrument [including computer- 
aided instruments utilising CHIs] in a consistent way — otherwise the instru- 
mentalist has no hope of learning how to perform on it musically.” Hence, 
control intimacy acts as a criterion for (musical) interfaces which are capable 
of responding “in consistent ways that are well matched to the psychophysi- 
ological capabilities of highly practiced performers.”® The idea of control inti- 
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macy does not take pre-existing intentions of the player into consideration, 
which should be conveyed by an interface designed as a transparent means. 
However, and crucially, the recent discourse on a transparent interface in 
which transparency is presumed to be one of the essential properties of an 
ex-pressive interface is hardly any different to a traditional concept of media 
which subsumes instruments, tools and devices as a transparent means of 
transmission of signals or representation of pre-existent entities. Should the 
transparency actually be seen as a property possessed per se by an interface 
which acts as a medium of an artistic experience? To address this issue, 
media theoretical discourses on transparency of a CHI will be surveyed. 


Transparency of a computer-human interface: 
From a New Media Theory point of view 


In different discourses on interface design and New Media Art, “trans- 
parency” has been discussed as a myth’ allowing the user to have a prefer- 
ably natural experience as ifit would be quasi non-mediated. In particular, 
the myth of transparency has been widespread with regard to virtual real- 
ity which is achieved through a three-dimensional representation providing 
the user a natural (three-dimensional) perspective of a computer-generated 
space. Immediacy is in this context a buzzword associated with transpar- 
ency. Theorists on New Media, Jay David Bolter and Richard Grusin, place an 
emphasis on “the logic of transparent immediacy” underlying virtual reality.’ 
Immediacy is here related to the perceptual illusion of an immersive “experi- 
ence without mediation”? which is called forth through a “disappearing act”!° 
of media. This myth of a non-mediated experience in computer-mediated 
communication also underlies research on virtual presence,'! the experience 
of being in a virtual environment, and on telepresence, the sense of being ina 
remote environment or virtual reality. Most research on presence is based on 
a realistic view of reality. Therefore the representation of a physical environ- 
ment as closely as possible serves as a condition for the reality created in a 
technologically mediated environment. This realistic representation, (referred 
to by Jonathan Steuer with his term “vividity” meaning a high degree of rep- 
resentation or “the representational richness of a mediated environment as 
defined by its formal features, i.e. the way in which an environment presents 
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7 On this see Bolter/Gromala 2003, pp. 48-56. 
8 Bolter/Grusin 1999, pp. 21-31 

9 Bolter/Grusin 1999, p. 23 

10 Bolter/Grusin 1999, p. 21 


11 This term is adapted from Sheridan 1992 to make an explicit reference of “presence” 


discussed for a computer-mediated experience. 


information to the senses”),!? is considered necessary for transparency of a 
technical apparatus used for virtual reality. Transparency of a medium is 
constitutive for virtual presence or telepresence based on the immediacy the 
user attains to, which the researchers on presence, Matthew Lombard and 
Theresa Ditton, characterise as “the extent to which a person fails to perceive 
or acknowledge the existence of a medium during a technologically mediated 
experience.”!? 

Jay David Bolter, however, directs our attention towards different strate- 
gies of interface design which include not only transparency, but also its 
opposites such as hypermediacy. Hypermediacy offers multiple signs of rep- 
resentation or reflectivity reacting to the user her- or himself situated in her 
or his surroundings and in the context initated by compelling experience 
mediated by means of an interface.'* Transparency is considered a strategy 
which can be chosen along the continuum consisting of manifold scales of 
hypermediacy. Hence, transparency cannot be regarded as a property pos- 
sessed per se by an interface. A property of an interface arises out of medial 
strategies of interface design which vary according to each artistic project and 
can be changed in the course of interaction. 

As opposed to interface design from an engineering science point of view 
(which aims in general at the illusion of transparent immediacy an inter- 
face offers), media art tends to direct the user’s/observer’s focus towards the 
media designed especially for any artistic purpose along with the palette of 
multiple meanings of the media emerging during the user’s/observer’s explo- 
ration. For media art, a medium can be both a means of, and at the same 
time a subject for artistic projects. In New Media Art, in which interactivity 
comes to the fore, a CHI designed and used for an artistic purpose serves 
as a medium which, however, does not remain static. It is assigned a tem- 
poral dimension due to the character of works of art based on a temporally 
expanded interaction space. Therefore a status change of the same interface 
as a medium relies on interaction actively explored by the user/observer. 

To make clear the idea that transparency of the media acting as a blind 
spot to bring the mediatised to the fore is not a property of a medium per se, 
in the following, some considerations on the performative logic of the medial 
are to be made, which are connected with the attempt to determine more pre- 
cisely the procedures of media in the light of a theory of transcriptivity.'® 
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12 Steuer 1992 
13 Lombard/Ditton 1997 


14 On the logic of hypermediacy see Bolter / Grusin 1999, pp. 31-44; on the strategy of 
interface design leading to reflexion see Bolter/Gromala 2003, pp. 62-65. 


15 This theory has been developed by the first author within the scope of the German 
Collaborative Research Centre Media and Cultural Communication (=SFB/FK 427). 
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The performative logic of the medial: 
Disturbance and transparency 


The term transcriptive refers to all infra- and intramedial procedures of 
cultural semantics which generate meaning through the mutual co-reference 
of various media or symbolic means of the same system.'® In the following, 
the term “transcriptivity” is to be specified more precisely against the back- 
ground of a media and communication theoretical model in the centre of 
which are the terms disturbance" and transparency"®. (At the same time, the 
term can certainly be understood against the background of some referential 
terms of New Media Theory, such as that of “cultural reconceptualisation”, as 
used by Manovich!?, or that of “remediation”, as introduced by Bolter and 
Grusin.?°) 

The model which I propose here assumes that communicative processes 
can be in at least two states of aggregation: (1) in that of non-disturbance, 
in which the medial (symbolic) means respectively used are not an issue as 
such, so that a direct “looking through”?! onto the semantics of the com- 
municated is possible. A communicative state of this kind can be described 
as a state of medial transparency. A second state is to be distinguished from 
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16 Jager 2002, in print (a) and (b). 


17 Interference is not understood as a ‘miscommunication’, as in the communication 
theoretical approach of Shannon, but as the constitutive impetus of communication; on 
Shannon see Shannon 1949; on the history of the impact of the Shannon/Weaver “flow 
diagram of communication” see also Schüttpelz 2002; for a more detailed discusssion of 
a media theoretical concept of interference, see Jager 2004. 


18 The concept of transparency originates from the semiological discussion of the close 
of the 18th and the beginning of the 19th century (here see also Jager 2004) and here is 
placed in a media and communication theoretical context. In the semiological tradition 
from which it comes, transparency does not mean that the medium becomes glass-like 
for the benefit of premedially existing contents, but that it virtually disappears with 
and in its function of constituting content. The concept of transparency therefore does 
not refer to the glassiness of the sign expression, but to the fact that the ‘transparent’ 
medium is dissolved in its content-constituting function, and so shifts receptive atten- 
tion from the mediation to that which is being mediated. In an approximately analogous 
manner to our own use of the term, Bolter and Grusin speak of “transparent immediacy” 
(Bolter/Grusin 2001, p. 21). 


19 Manovich 2001, p. 47: “In new media lingo, to “transcode” something is to translate 
it into another format. The computerisation of culture gradually accomplishes similar 
transcoding in relation to all cultural categories and concepts. That is, cultural catego- 
ries and concepts are substituted, on the level of meaning and/or language, by new ones 
that derive from computer’s ontology, epistemology, and pragmatics. New media thus 
acts as a forerunner of this more general process of cultural reconceptualization.” 

20 Bolter/Grusin 2001, p. 45: “(...) we call the representation of one medium in another 
remediation, and will argue that remediation is a defining characteristic of the new digi- 
tal media.” In the same vein, for instance p. 55: “It would seem, then, that all mediation 
is remediation. (...) No medium, it seems, can now function independently and establish 
its own separate and purified space of cultural meaning.” 


21 On the distinction between “looking through” and “looking at” see Bolter/Grusin 
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this, (2) the state of interruption of the transparency mode by an interactant 
for the purpose of fixing the attention on sign sequences communicated, and 
their mono- or interactive processing on a semantic stage for negotiating. 
A communicative state of this kind can bring about a “looking at”? of par- 
ticular, topicalised sections of medial performance in their material presence, 
because these are detached from the communicative process and become 
the object of transcriptive attention. I would like to describe this communi- 
cative state as the state of disturbance. Disturbances mark those moments 
of medial communication in which the medium itself becomes the object of 
communicative attention. 

Disturbance and transparency are to be understood as two polar func- 
tional states of medial performance which are constitutively inscribed in the 
process of transcription. Transcription could then be described as the transi- 
tion from disturbance to transparency, from the decontextualisation to the 
recontexualisation of the signs/media on which the focus is placed. While 
disturbance as the respective point of departure actuates the transcriptive 
process of remediation and brings the medium as a (disturbed) operator of 
sense into the focus of attention, transparency can be seen as the state (in 
the process of medial performance) in which the respective medium disap- 
pears or becomes transparent with respect to the content which it mediatises 
(distributes, archives, constitutes). Disturbance and transparency therefore 
mark two modes of visibility which are in general mutually exclusive: the 
visibility of the medium and that of the mediatised. The invisibility (transpar- 
ency) of the sign/medium virtually allows the undisturbed ‘realism’ of the 
mediatised, while the becoming visible of the medium, i.e. the irritation of the 
habitualised contexts of use, indicates the looming crisis of the ontological 
illusion which is then withdrawn from the mediatised objects. Realism is, as 
Goodman notes, relative to the media: “(...) it is determined by the system of 
representation standard for a given culture or person at a particular time,”’* 
or by the medial dispositive in which the communication respectively takes 
place. The realism with which symbolic means carry out representation”* 
is higher, the more familiar (the more transparent) the means selected are. 
Medial displays seem realistic to us when “practice has rendered the symbols 
so transparent that we are not aware of any effort, of any alternatives, or of 
making any interpretation at all.” In communicative states of this kind, the 
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22 Bolter/Grusin 2001, p. 41 
23 Goodman 1976, p. 37 


24 For Goodman representation does not mean “mirroring”. (Pictoral) representation 
and (verbal) description are kinds of denotation: “Representation is thus disengaged 
from perverted ideas of it as an idosyncractic physical process like mirroring, and is rec- 
ognized as a symbolic relationship that is relative and variable” (Goodman 1976, p. 43). 
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mediatisation of the real is “... obscured by our tendency to omit specifying 
a frame of reference when it is our own,” or - formulated differently - when 
it is one of our own present (undisturbed) language games which is in use 
as a symbol system. Only “changes in representational practices”,? in other 
words effects of disturbances, only the erosion of habitual contextual frames, 
allow the medial relativity of the real and hence the symbolic representation 
system to become visible as a “way of worldmaking”” again.” Crises of this 
kind are natural as disturbances of semantic equilibrium, not just epochal 
events which every once in a while distress cultural viewpoints and world 
views. They are first and foremost, common transitional stages which medial 
processes pass through according to their own logic of transcriptivity until 
communication in turn enters into phases of transparency. 

Disturbance is then taken to mean that state in the process of a com- 
munication which has the effect that a medium (operatively) loses its trans- 
parency and is perceived in its materiality. Transparency in turn means that 
state in which communication is not ‘disturbed’, and so the medium is not 
in the focus of attention as a medium. Transparency can be understood for 
instance, in the sense in which Luhmann assumes that in the interdependent 
relationship of medium and form the form is visible and the medium remains 
invisible.?° If one were to transfer the disturbance-transparency model onto 
Luhmann’s medium-form distinction, disturbance would be the state of a 
communication in which it is not the form which is observed through the 
(invisible) medium, but the “contingency of formations”?! or “the free capac- 
ity of the medial substrate to make ever-new couplings”? that would be 
observed in the medium. One could also - following on from Edgar Rubin and 
Marshal McLuhan - say that the medial process in the state of disturbance 
brings the medium into the focus of attention as a figure, while it recedes 
into the background in the state of transparency: “(...). All cultural situations 
are composed of an area of attention (figure) and a very much larger area of 
inattention (ground). The two are in a continual state of abrasive interplay, 


26 Goodman 1976, p. 37 

27 Goodman 1976, p. 39 

28 Goodman 1978 

29 For acritical discussion of the media theoretical implications of Goodman’s symbol 
theory, see Mitchell 1994, pp. 345-362. 


30 On this see for example also Luhmann 1997a, pp. 190-202. For instance, the fol- 
lowing is noted with respect to the perceptive media: “We do not see the light, but things 
(...). We do not hear the air, but noises.” (Luhmann 1997a, p. 201; translated quoting); 
on this see also Kramer 1998. Following on from Luhmann, she formulates “that wher- 
ever we encounter media, we do not perceive media themselves, but only forms” (Kramer 
1998, p. 76; translated quoting); also Kramer 2001, p. 157: “Moreover, the form is visible 
— the medium, on the other hand, remains invisible.” (translated quoting) 


31 Luhmann 1997b, p. 168 (translated quoting) 
32 Luhmann 1997a, p. 200 (translated quoting) 
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with an outline or boundary or interval between them that serves to define 
both simultaneously.”?® If Alfred Schütz were to be brought in, we could also 
describe the state of disturbance as the becoming relevant of the medium, 
and the state of transparency as its return to the mode of familiarity.” What 
could be even more illuminative for the unravelling of the processual comple- 
mentarity of disturbance and transparency than the terminological analogies 
provided by Luhmann, Rubin, McLuhan and Schütz is a pair of terms which 
has been talked about in the discourse of analytical philosophy in a con- 
text having to do rather with the logic of research, namely the terms implicit 
and explicit knowledge.” In a broad sense transcriptive processes, insofar as 
they are understood as processes which move out of disturbance and into 
transparency, can then be understood as processes of “expressing”: “(...) as 
a matter not of transforming what is inner into what is outer but of making 
explicit what is implicit.”°° If - as Brandom formulates - “what is expressed 
must be understood in terms of the possibility of expressing it,”*’ it is one 
of the constitutive conditions of implicit semantics that it - in the case of 
communicative disturbances — must be able to be made understandable in 
explicative (transcriptive) processes. Hence while in the state of medial trans- 
parency semantics is processed in the form of silent knowledge, explicative 
(transcriptive) actions are required when the implicit in one form or another 
becomes problematic or the subject of attention, and with this the medium 
as such comes to the fore. Transcriptions are therefore to this extent explica- 
tions which under particular communicative discursive conditions are once 
again transferred into the status of implicit, i.e. silent knowledge. If this kind 
of precondition is assumed, then the medium is the mediator** of something 
which — depending on the aggregate state of the communication — changes 
between figure and ground or between relevance and familiarity. And it is 
precisely this continuous changing which allows the medium to be more than 
an expression and a mediator of something internal: namely the explicator 
of something implicit which, by being explicated, changes its epistemic sta- 
tus to such an extent that one could say, in a sense that the implicit is not 
only expressed, but also constituted by its explication. The medium is then 
the (performative) place, the place of processing at which implicit semantics 


becomes explicit, only to - in the case of ‘undisturbed’ communication — enter 
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33 McLuhan/Powers 1989, p. 5; on this see also Fohrmann 2008. 

34 Schütz 1971; on this see Jäger 2001. 

35 On this see for example Hare 1974; Polanyi, 1966; Brandom 2000. 
36 Brandom 2000, p. 8 

37 Brandom 2000, p. 9 


38 “Mediator” in the sense of Engell und Vogel, who understand the medium as “the 
middle and the median, the mediation and the mediator” (Engell/ Vogel 1999, p. 9; trans- 
lated quoting). 
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into the state of the implicit once again, by which process the medium disap- 
pears behind the semantics which it helps to organise (without being absent). 
Explications are to this extent processes of (disturbance-induced) focusing 
and concentrating on media (and their implicit semantics) in the interest 
of creating (explicative) semantic effects which, when they occur, push the 
medium out of the focus of attention once again. They are processes of recur- 
sive self-processing, i.e. the application of communication to the effects of 
communication.’ I would like to call this feedback movement which opera- 
tively determines the processes of media systems recursive transcriptivity.* 
It is this process of recursive transcriptivity through which the processes of 
disturbance and transparency are connected with each other, through which 
— under different medial and communicative conditions — the processes of 
cultural semantics are kept going, i.e. are updated through alternating stages 
of stabilisation and irritation in fragile states of equilibrium which can be 
disturbed at any time. In contrast to many media theoretical positions, the 
model proposed here does not assume that transparency can be seen as a 
constitutive property of the medial. That media tend to “become virtually 
imperceptible, anaesthetic” with respect to what they mediatise,*! that they 
remain invisible under certain conditions®?, because - as Fritz Heiders for- 
mulated in 1926 - a true medium is only one through which “one can see 
through ... without obstruction”,*® does not refer to a quasi-ontic feature of 
the medial, but to a particular stage which medial processes pass through in 
the process of recursive transcription. So the thesis which is advanced here is 
that the transparency of the medium is not a ‘property’ of the medium, but an 
aggregate state in which the mediatised semantics as silent knowledge is not 
communicatively ‘disturbed’. Just as conversely, disturbance is not a para- 
sitical defect of communication, but that aggregate, communicative state in 
which the sign/medium becomes visible as such, and hence semanticisible. 
In other words, disturbance is that state which is always connected with the 
need for remediation, i.e. transcription. 
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39 See here Luhmann’s concept on the recursive “individual behaviour” of systems: “It 
[the term “individual behaviour”] refers to a stability reached in the recursive procedure 
of the application of procedures to the results of procedures.” (Luhmann 1997a, p. 213; 
translated quoting) 
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41 Engell/Vogel 1999, p. 10; see in particular footnote 15. 
42 Kramer 2001, p. 157 

43 Heider 1959, p. 3 
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Transparency and opacity 
of an interface in New Media Art 


Applied to our research area of New Media Art, we can behold a performa- 
tive logic of the media underlying an artistic interaction, which is based on 
the oscillation between disturbance and transparency of a computer-human 
interface. The act of recursive transcription which is constitutive for transpar- 
ency of the medium emerging as an effect during a medial performance can 
be illustrated in cases of interaction between bodily movement and sound. 
In artistic interaction mediated by an interface translating bodily movement 
into sound, the interface operationally loses its transparency in favour of the 
processing of bodily-based expressions of generated sounds. In this state, the 
interface becomes - with its physical materiality - opaque. Every new kind of 
interface gives rise to partially or completely unfamiliar relationships between 
bodily movements and sounds. This allows a user to explore explicitly how 
motor and auditory feedback may be coupled. Dealing with a new interface 
hence includes a self-perceptive moment in the progress of artistic produc- 
tion. Each act within the interactive situation seems to begin with a kind 
of disturbance that redirects one’s attention to the opacity of the interface. 
Opacity first comes to the foreground so that a user may generate expressive- 
ness, appropriating the procedures of intermedial transcription, as an effect 
which, once achieved, shifts the interface — without making it become really 
absent - out of the focus of attention. In this state of transparency, the self- 
significance of an interface becomes absent. Thus, the mediating aspect of 
interface technology does not seem to consist in the function of an interface 
as a means of expressing or transferring a user’s pre-existent intentions or 
states. Rather, the medial operation of oscillation between transparency and 
opacity renders an interface capable of problematising an implicit artistic 
intention or meaning and constituting it as an effect. 

Let us have a look at some concrete strategies of the design of interfaces 
mediating bodily movements and sounds. One of them is oriented towards 
a model of physical-acoustic musical instruments. This aims to simulate or 
extend traditional physical-acoustic musical instruments by digital technolo- 
gies. In these strategies, a bodily movement acts as a kind of trigger and con- 
troller for digital sound synthesis. The principle of digital (algorithmic) sound 
generation is characterised by a decoupling of the sound generation mecha- 
nism (synthesis algorithm) from its control device — this is substantially dif- 
ferent from the mechanism of sound generation underlying a physical-acous- 
tic, musical instrument in which sound generator — e.g. violin strings — and 
controller — e.g. violin bow - are closely coupled. In digital musical instru- 
ments, a mapping from the input parameter of bodily movements as a control 
parameter into an output parameter used for algorithmic sound synthesis, 
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which is called gesture mapping (Fig. 1), therefore plays an important role for 
a (re-)coupling of sound control and generation mechanisms. 
Such an artifi- 


cially enacted relation gesture mapping 


between bodily move- 
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constituted by tran- sound 


synthesis 
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mapping algorithms. 


Hereby, transcription Fig. 1. Gesture mapping 
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bodily movement and sound. When an interface is in a state of opacity, a user 
is required to ascertain this intermedial coherence which becomes evident in 
an action-perception loop taking place during an interaction. 

At the outset of dealing with a musical CHI, a user/musician is able to 
recognise what strategies of remediation of a traditional musical instrument 
have been used in the design of the CHI: highlighting, refashioning or absorb- 
ing it.** The design of the series of MIDI (Musical Instrument Digital Interface) 
controllers — for instance, MIDI keyboard, MIDI string and wind controller 
- simulating the controller of physical-acoustic, musical instruments in con- 
nection with synthetic instrument sounds, persues a strategy highlighting the 
musical instrument. Another group of musical interfaces called augmented 
musical instruments or instrument-inspired gestural controllers are designed 
with strategies of reshaping musical instruments. 

Let us take the overtone violin developed by the composer Dan Overholt 
as an example of an augmented musical instrument. This controller is capable 
of preserving the traditions of violin bowing technique in dealing with the 
strings (albeit six strings instead of four). A set of embedded sensors added to 
this controller, however, afford extended or even new possibilities for musical 
control. Violin playing techniques captured for instance by optical pickups 
sensing the vibrations of the strings are used for the creation of expressive 
nuance of sounds. Other musically relevant gestures captured by diverse ges- 
ture-sensors are used to add new control dimensions to the instrument.” 

As a result, musicians can play this augmented instrument with violin 
playing technique as is usual and at the same time explore a new playing 
technique. Such a musical interface might become transparent while musi- 
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cians are mastering the instrumental technique necessary for the combina- 
tion of the usual and novel control dimensions. The “playing” of augmented 
musical instruments therefore requires an intensive time-consuming exer- 
cise, comparable to dealing with traditional musical instruments. 
Instrument-inspired gestural controllers featuring a strategy of reshap- 
ing a musical instrument allow musicians to use a more or less traditional 
instrumental technique to generate and modify sound events. However, the 
sounds that actually emerge may likely have little to do with the instrumental 
gestures normally associated with their production. An example of instru- 
ment-inspired, gestural controllers is the violin-inspired controller BoSSA 
(Bowed-Sensor-Speaker-Array) devel- 
oped by the composer Dan Trueman 
and Perry Cook (Fig. 2). This instru- 
ment consists of the violin’s physical 
performance interface and its spatial 
filtering, audio diffusor, yet possesses 
neither the resonating body nor the 
strings. Instead it is equipped with a 
bow interface extended with pressure, 
force resistance and accelerometer 
sensors and virtual strings consist- 
ing of sponge divided into four parts 


which can be bowed.* In this way, 
Fig. 2. Dan Trueman is playing the musicians can use a set of possibili- 
eer ties of bowing techniques familiar to 

them. But gestural data captured by 
different sensors while playing this interface can be mapped arbitrarily to 
any musical parameter so as to give rise to a change of loudness, vibrato or 
timbre, for instance. 

As a consequence, musicians are confronted with the opacity of the con- 
troller at the outset far beyond their expectations which have been caused by 
the possibility of using traditional, gestural techniques. The novel relationship 
between instrumental gesture and sound events triggered by this gesture, 
however, challenges a musicians’ available image underlying the playing of a 
musical instrument. This disturbance gives rise to a reprocessing of interme- 
dial relationships between gestural activities used for instrument technique 
and auditory as well as tactile perception, in the course of which a new image 
of playing technique is constituted, replacing a previous one. This process of 
constitution of a (mental) image is due to intermedial transcription which is 
based on cultural semantics associated with instrumental techniques and 
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used tone systems. Transparency of such 
an instrument-inspired gestural controller 
is achieved when the interaction renders a 
musician capable of becoming habituated 
and of feeling musical expressiveness in 
dealing with the interface. 

Alternate controllers which do not 
have any affinity to a traditional musi- 
cal instrument allow the user/performer 


to develop her or his own strategy which 
mediates bodily movements and sounds, Fig. 3. Michel Waisvisz is play- 
ing the second prototype of 
The Hands. Courtesy of Michel 
Waisvisz 
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since habitual behaviour adapted from an 
old medium cannot be applied. But some 
musicians tend to design such controllers 
as instrument-like to offer a gesture mapping from musically meaningful ges- 
tures intentionally produced, into comprehensible and reproducible musical 
events. A well-known alternate controller The Hands (Fig. 3) developed at 
STEIM (Studio for Electro-Instrumental Music) since 1984 is a prominent 
example for instrument-like, alternate controllers. Although the shape of this 
interface does not have any similarity to a traditional musical instrument, 
it allows a musician to perform with virtuosity. A lot of alternate controllers 
which have been used in less musically oriented, media performances pur- 
sue a strategy of absorbing a musical instrument, so that each user — not 
only a musician - is required to (re)build her or his own image with regard 
to the intermedial relationship between bodily movement and sound. In the 
stage of getting to know such an interface, the medium is fully in the focus of 
attention. In this state of disturbance of an interface, a transcriptive process 
begins with problematising an implicit image and explicating it while dealing 
with a medial trace in the process of artistic interaction, in which the medium 
becomes ultimately transparent and an image explicitly explored becomes, 
in turn, implicit. An implicit knowledge of how to deal with each new inter- 
face is generated as an effect of embodied interaction guided by the match 
between bodily action on the one hand and auditory and tactile perception on 
the other. The user’s/observer’s active involvement in an unknown (artistic) 
world offered by New Media Art therefore acts as a condition for the genesis 
of an artistic experience which then comes into the focus of attention as a 
figure, pushing the interface into the background. 

An artistic experience, which is often related to expression on the one 
hand and to impression on the other does not seem to be strictly separable 
into productive and receptive experience in New Media Art, where artists often 
act as observers of constituted works of art and vice versa at the same time. 
A person who enters into a computer-generated world acquires an artistic 
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experience via a coupling of activity and passivity, production and perception, 
motor and sensory processes. In this way, coupled acts of sensory and motor 
processes have an impact on (completion of) implicit artistic expressiveness 
experienced by the interactant, while explicitly exploring the strategies of an 
interface, artificially mediating action-perception loops. 

The mediality of an artistic CHI therefore consists in a reshaping and 
constitution of artistic experience which is initiated by a disturbed, opaque 
state of an interface as a medium. Each strategy of an interface mediating 
and modeling a relationship between bodily movement and sound proves to 
be based on recursive transcription which attracts one’s attention towards 
explicating processes of artistic experience to attain to implicit aisthesis. In 
this way, interface technology of mediation, oscillating between transpar- 
ency and opacity of the medium, accentuates the mediality of (intramental) 
aisthesis even by modeling intermental processes taking place in embodied 
and situated interaction such as artistic human-robot interaction, which 
require transcriptive remediation to constitute a cultural semantics of artistic 
experience. 
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1. Distributed agency: 
A concept beyond human action and technical means 


Usually, the action is where the humans are. Action means moving the 
body, making something, showing initiative, bringing about an alteration by 
force, and expressing oneself thereby. Action becomes particularly visible 
when there is unexpected reaction to something or resistance to somebody’s 
will. In the humanities and social sciences, action is closely associated with 
the anthropological concepts of man the artist and tool-maker or the speaker 
and symbol-communicator.! Human action — defined to be intentional and 
creative — is often sharply distinguished from animal behaviour, which is 
characterised as instinct-driven and only tool-using, and from machine oper- 
ation that is described as a repetitive and pre-programmed activity. If we con- 
tinue to define action by the demanding features of intentionality, rationality 
or reflexivity that are attributed to humans only, then — no wonder - all other 
uses of the term “action” in everyday life and actual technological develop- 
ments would be only metaphors or even categorical mistakes. In this case we 
would miss and misunderstand the massive changes in intelligent machine 
design and interactive media use that open up Pandora’s box filled with thou- 
sands of agents. These software or hardware agents equipped with belief, 
desire and intention algorithms are able to take part in manifold actions and 
even to change their action programs by case-based learning. Certainly, they 
are different from human actors, but they are also different from classical 
machines and media. Both features, their particular capacities of being active 
and interactive and their growing population in everyday gadgets and in the 
worldwide web of the internet, justify the undertaking which has been made 
in the following, to develop a more symmetrical and sophisticated concept of 
agency. 

What are people talking about when they use the word ‘action’ in every- 
day life? Do the youngsters still mean good old human action, when they are 
acting in videogames inducing an avatar to follow and fight other creatures 
only by button-pushing? It is evident that button-pushing in this case is not 
the one single and simple instrumental action of fighting with swords, but 
one activity under many others: It activates a cascade of programs which 
themselves activate characters that show contingent action in a virtual action 
environment. 

The players surely know the fundamental difference between the other 
human actors and the artificial agents in the game; but they are more inter- 
ested in the interactivity and the particular high level of agency that they 
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experience during their interaction with both kinds of partners or adversar- 
ies: the humans and the agents. 

We learn something about the meaning of action when we listen to peo- 
ple talking about the genre of action films. They do not only mention the 
human actors who are in states of super-activity like running, jumping, or 
shooting and who are entangled in highly interactive situations like chasing 
one another or fighting with one another. Action includes more than human 
bodies in interaction. It is closely connected with the activities of high-speed 
vehicles, explosives and firing weapons as we know so well from James Bond 
films. Action of this special kind emerges from accelerated sequences of action 
of all kinds of acting units. The impression and fascination of action is finally 
produced by the many interactivities between the mixed agencies, not by the 
human interaction alone. 

Actually, computer and media scientists use the vocabulary of human 
action when they describe the features of new technologies. Are software 
agents, for instance, really acting like human actors, when they ask the user 
for tasks, when they cooperate and compete with one another in the artificial 
society of agents, and when they assist persons in their daily actions of sort- 
ing out e-mails, searching for optimal traffic connections, looking for best 
prices, booking tickets and buying investment papers? Is it correct when 
interface designers speak about Human-Computer Interaction (HCI) and 
students of Distributed Artificial Intelligence call their programs ‘agents’ or 
‘multi-agent-systems’ because they are constructed with the explicit inten- 
tion to act like a person who is acting in the name of an other person? Action 
can be composed of different acts, and some can be delegated. Collective 
actions can be unified in a corporate actor like an organisation. They can be 
divided between principal and executive agents.’ If the actions are distributed 
between human actors and nonhuman agents,? why should we not treat this 
‘hybrid constellation’ as a particular kind of a collective actor? 

Answering the main question ‘where the action is’ actually seems more 
complicated than before. The introductory considerations have alluded to 
four relevant changes in the sphere of human-technology relations that call 
for some conceptual revisions: 


- The number of acting units and the kinds of action are increasing for 
the first time since modernity and enlightenment successfully dimin- 
ished it by banning moving objects and talking trees, inviting nymphs 
and punishing gods, speaking oracles and helpful angels out of the 
sphere of action into the world of fetish and fiction. 
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- Instrumental actions between active people and passive objects are 
turned more and more into relations of interactivity between two het- 
erogeneous sources of activities. The analysis and design of these rela- 
tions require a more balancing approach of interactive contingency 
than a hierarchical one of instrumental causality. 

- Actions are fragmented in many pieces and delegated to myriads of 
pro-active and cooperative agents on the back stage where they per- 
form parts of the action by mimicking human agency and interper- 
sonal interaction. 

- Actions emerge out of complicated constellations that are made of 
a hybrid mix of agencies like people, machines, and programs and 
that are embedded in coherent frames of action. The analysis of these 
hybrid constellations is better done with a gradual concept of distrib- 
uted agency than with the dual concept of human action and machine’s 


operation. 


In this paper it is argued that the advanced technologies take part in the 
course and constellation of human action and that they do this with real 
effects, not only metaphorically. The first part starts with the search for a 
useful concept of agency that enables the researcher to describe and classify 
all activities that contribute to the performance of an action. The concept 
shall include different levels of human agency as well as different levels of 
technologies in action (2). The following chapter treats the consequences that 
these activations of technologies have for the human-technology relation. If 
technologies change their role from passive means into agents and media- 
tors, then the narrow concept of instrumental action should be replaced by a 
broader concept of inter-agency (3). This part of the paper culminates in the 
presentation of a gradual model of agency that can be used to describe and 
distinguish between different levels and grades of action without any regard 
to the ontological status of the acting unit, be it human-like or machine-like 
(4). 

In the second part of the paper the question ‘What is the adequate unit of 
action?’ is answered. It starts with a thought experiment about the question: 
Who is really flying the Airbus? We learn from both views, the humanist’s and 
the technologist’s one, that what is usually called action, such as flying 240 
tourists to Tenerife airport, consists of many distributed actions that have to 
be coordinated by social organisation or technical configuration (5). The con- 
cept of distributed agency is spelled out in three steps: It presupposes many 
loci of agency, not one actor (5.1). It declares the hybrid constellations made 
of the mixed human and material agencies to be the adequate research unit, 
and not solely the homogeneous social organisations or the technical configu- 
rations (5.2). Finally, a third mode of integration called ‘framed interactivity is 
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elaborated that may emerge between the hierarchical mode of master-slave- 
relation and the open mode of autonomous systems (5.3). 


2. Technologies in action: From artifacts to agents 


Human action and technological operation belong to two different worlds: 
the realm of freedom and the realm of forces. Following Kant’s definition, 
human action is characterised by its moral autonomy from external forces 
and laws. Although humans are subjected to these forces, they have the 
capacity (free will) to give themselves the rules of action that may become the 
general maxims for others, too. Referring to Reuleaux’s definition, machines 
follow the very idea of forced movements. Heteronomy is the characteristic 
of von Foerster’s “trivial machines” that are completely determined systems.* 
The dichotomy of tool-maker and artifact is completed by the dichotomy of 
rule-making and rule-following. 

This fundamental dichotomy may be helpful to divide between the onto- 
logical spheres of morality and causality. But it should not be applied to 
our questions of empirical changes and practical consequences. If we want 
to analyse the gradual changes of advanced technologies, the qualitative 
changes of the interaction between people and technologies, and even more, 
the re-configurations of the hybrid constellations from which action emerges, 
then we have to overcome this dual concept of action and operation. Thus we 
start with a symmetrical concept of agency that permits us to describe and 
classify what could be meant by the feature ‘in action’. On this low level, we 
look for features of self-movement, activeness and self-acting. 

How can we decide whether advanced technologies have changed and 
in which aspects? Let us take the five aspects that are often used in the 
engineering literature: technology as a motor / driver (“Motorik”), as an actua- 
tor (“Aktorik”), as a sensor system (“Sensorik”), as an information proces- 
sor (“Informatik”), and as a communicator (“Telematik”). With respect to the 
aspect of motion, we can state that the gadgets and machines have gained 
higher degrees of self-movement: from one central stationary steam engine 
towards distributed systems of many engines powered by electric drive, from 
externally driven carts and coaches to self-driven vehicles, called automo- 
biles. Under the aspect of acting and working, we make out a strong drift from 
crafted tools through mechanical machines to automatic systems. 

The next three aspects seem to be of critical importance for the level of 
technologies which are subsumed under the label of “smart machines”, “intel- 
ligent systems”, “new electronic media”, or “high technology”.® Regarding the 
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aspect of context-sensitivity, we actually realise a strong tendency away from 
systems that are completely blind to ones that are equipped with a feed- 
back mechanism, all the way up to highly sensitive systems that are able 
to realise situations and to adapt their action to changing environments. 
The greatest steps in the direction of activating technical objects have been 
made with respect the aspect of information-processing: Looking backward, 
we reconstruct the movement as a loop from hard-wired tools and machines 
whose activity plans are incorporated in the design of the artifact, via flexible 
machines that are programmed by cards and records towards highly autono- 
mous systems that strongly self-control their activities by nested systems of 
computer programs. Last, but not least, the aspect of communication between 
objects has emerged. Communication about the state of the machines’ activ- 
ity has been the task of people observing them at the work bench or in the 
office of the factory supervision. The direction is now inverted: The machines, 
the gadgets and even the products themselves observe the states, places and 
times of their actual activity and communicate them to people and also to one 
another via cable (Internet) or radio frequency (RFID). 


ASPECTS FROM CHANGE TOWARDS 
LLC DD ELLE 
Motor stationary gadget >>> mobile agent 

Actuator passive instrument >>> pro-active agent 

Sensor blind machine >>> context-sensitive agent 
Processor hard-wired artifact >>> programmed agent 
Communicator single apparatus >>> cooperating agent 


Fig. 1. Aspects of technological change 

The current advanced technologies show signs of increased self-activity 
within each aspect. As they are human-made technologies, they remain arti- 
facts. However, they loose their passive, blind, and dumb character and gain 
the capacities to be pro-active, context-sensitive and co-operative. Insofar as 
the technical artifacts have been put into action by these changes, it is justifi- 
able to define them as agents. 

What are ‘agents’? From a technological view, agents are particular com- 
puter programs. They are written with the intention that software agents can 
execute actions like human agents. This means that actions are delegated to 
them. The agents divide and delegate the action among other agents. They 
cooperate with one another, thereby moving, taking the initiative and address- 
ing others. They coordinate the cooperation themselves and communicate the 
result of their activities to the human user. In a seminal text on intelligent 
agents, the main characteristics are presented as relative “autonomy”, a par- 
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ticular “reactivity” to the environment, “pro-activeness”, and “sociability”.® 
From a sociological view, agents are persons who act in the name of a princi- 
pal, e.g. the owner of an enterprise or as an informant of a party in a strategic 
spy game.’ The business and the secret service agent are bound to the gen- 
eral aims of the principal, but they are free to choose the adequate actions. 
Their actions are not blind executions of the principal’s will. Agent-oriented 
programming and the design of architectures for multi-agent-systems follow 
this social concept of an agent and take over other mechanisms of society 
like cooperation, competition, trust or community in order to establish more 
flexible systems of distributed artificial intelligence. The up to now dominant 
design of a master-slave architecture is slowly being replaced by open sys- 
tems of distributed and cooperating agents. The higher grade of activeness 
given to the software agents motivates the software engineers and the system 
designers to transfer those social and sociological concepts which have been 
proven as successful mechanisms of coordination.® 

Technologies are changing on the level of technical systems, not only 
as concrete tools, machines, media, and sign processors. They show higher 
levels of complexity, they are more heterogeneously combined, and they are 
more complicatedly nested with one another. A review of the advanced tech- 
nological and media systems reinforces the impression of a radical change in 
quality, not only in quantity and diffusion of technical objects. The Airbus is 
highly complex in a different way than a cathedral that is also made of mil- 
lions of stones, glass pieces, and thousands of fixed relations between them, 
or than a Cadillac car in the fifties that is assembled out of thousands of 
exchangeable parts and has hundreds of variable relations between them. 
The cathedral and the Cadillac, however, combine heterogeneous materials 
and technologies, but the construction of an Airbus requires the integra- 
tion of much more diverse technologies in an incomparable way. Especially, 
the embedding of so many different programmed physical and information 
systems in one plane produces the system’s opacity that favours the inter- 
pretation of being confronted with an autonomous being. Stanley Kubrick 
clearly demonstrated this strange feeling in his “Odyssey in Space” when the 
computer system HAL, which was a part of the automated space ship, had 
to cope with contradictory rules in his program, then resisted human control 
and started to follow its own rules of action. 

It is precisely to escape such fantasies of autonomous action on the one 
hand and the stubborn notion that technologies do not show any sign of 
agency on the other hand, that a more differentiated approach to the problem 
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of technology in action should be developed. A scale with five levels of agency 
is presented here which may be seen as a first step on this route. The prin- 
ciple of its construction refers to the performance of technical objects and 
systems, not to their function. It also refers to the above-mentioned aspects 
and their interrelatedness. Examples from different technological domains 
are given for reasons of understanding. This scale is designed to raise aware- 
ness about different levels of agency and can be used for descriptive and 
classificatory reasons. 


LEVEL OF AGENCY DESCRIPTION EXAMPLES 


0 0 zZ 


Passive Instruments completely Hammer; Punching card 
moved from outside 


Semi-active Apparatus with one aspect Machine tool; 
of self-acting Record-Player 
Re-active: Systems with feedback loops Adaptive heating system 
Pro-active: Systems with self- Car stabilisation; 
activating programs Help agent 
Co-operative: Distributed and self- Mobile robots; 
coordinating systems Smart Home 


a gie 

It is not so easy to give examples that are typical of the particular level. 
The position in the scale depends on the precise description of the equip- 
ment and the connectedness between its parts. A brake can be a simple tool 
that functions mechanically. It can also be activated by a little motor; then 
it changes to the level of a semi-active hydraulic machine. When the brake 
is connected with a feed-back measurement instrument, it then operates on 
the level of re-activeness. Actual brake systems in the ICE or TGV trains 
are to be allocated on higher levels: They are pro-active, because they start 
their action themselves after having monitored and computed critical dates of 
inner and outer states. When there is also communication between the brake 
systems at the different wheels, then we can speak of a distributed and co- 
operative system. What can be learnt from this example? New insights cannot 
be gained from talking about agency on the first two or even three levels. It 
is completely sufficient to use the mechanical vocabulary of operation and 
determined movements. When the parts of a technical system, however, can 
behave not only in one pre-fixed way, but more flexibly, when the interaction 
with other parts or the interaction with the environment changes the behav- 
iour, and when some parts actively search for new information to select their 
behaviour and even more to change their pre-given frame of action, then and 
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only then does it make sense to use the vocabulary of agency and interaction 
in the world of objects. 


3. Types of Inter-Agency: 
From instrumentality to interactivity 


What makes it happen that a move, a behaviour or any other activity is 
recognised as a significant gesture or a meaningful action? How do we know 
whether a winking eye is only a body reflex or an intended signal to be willing 
to flirt? How do we know whether a flashing sign on our screen of our PC is 
a mechanical mistake, a routine recommendation to continue writing at this 
point or the triggered sign of an unexpected spy software? If one follows the 
social theory of pragmatism, the answer would be: One has to observe the 
sequence of three acts and relate them with one another as a circle of inter- 
action. It is only at the end of this threefold interaction process that one can 
attribute the label causal effect, instinctive behaviour or meaningful action 
to the initial move. 


Act 1: Winking eye >>> Act 2: Winking back >>> Act 3: Winking and laughing 
Act 1: Winking eye >>> Act 2: Winking back >>> Act 3: Looking away 
Act 1: Winking eye >>> Act 2: Looking away ashamed >>> Act 3: Also looking away 


These three sequences demonstrate that the meaning of the winkling eye 
in act 1 can only be ascertained after the next two acts: In the first line, 
the additional laughing completes the interaction circle and makes the first 
winking into a significant part of a social interaction called flirting. Act 3 in 
the second line constitutes the same meaning, but a different attitude to it, 
namely not being interested in flirting. The two consecutive acts of looking 
away in the third line seem to constitute a different meaning to the winking 
eye in act 1: It is an illness of nerves for which one does not want to stigma- 
tise the person. What is important to note in the context of our argumenta- 
tion, which is the central message of pragmatical interactionism is the fol- 
lowing: The interactions observed and practised between the units of agency 
are what make critical differences and constitute the relevant meanings, not 
the individual act. 
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This approach is usually applied to interpersonal interactions between 
human actors only, as in the case above. However, one can find some hints in 
the literature that the approach can also be transferred to relations between 
people and objects.’ For systematical reasons and for our particular purpose, 
three types of inter-agency should then be distinguished: 


- Interaction between human actors, 
- Intra-activity between technical agents, and 
- Interactivity between people and objects. 


Interpersonal interaction constitutes the social world of ‘inter-subjectiv- 
ity’. It is populated by human actors, expectancies and communications; 
it is structured by institutions, social systems and cultural meanings; it is 
the classical subject of the social sciences. Technical objects are principally 
excluded from this sphere of pure sociality; they figure either as neutral 
means for purposeful action, or as irritating objects from outside the society, 
or alternatively they are interpreted as mere carriers of meanings. 

Intra-activity is quite an unusual term: In analogy to the relations between 
people it can be confined to the relations between objects, especially between 
technical agents. It constitutes the material world of ‘inter-objectivity’.!° In so 
far as the objects show low levels of agency — according to our scale - and in 
so far as they are strongly coupled in linear, sequential or otherwise aggre- 
gated ways, one does not need to open the black box to study the internal 
operations. If, however, they display higher levels of agency and show more 
loosely coupled relations between the units, as in the cases of complex and 
high-risk systems or in cases of distributed and multi-agent systems, one 
should also follow the activities of the objects and describe their intra-activ- 
ities. Otherwise one could not understand the differences which come up 
when people get into use relations with these kinds of technical systems. For 
it makes a difference whether people encounter an encapsulated system or a 
cooperating ensemble of agents, a hierarchical fixed order or an open network 
with case-based learning. 

Interactivity is the term that is reserved for the cross-relations between 
people and objects. It belongs to the hybrid world of “interfaces”, “human- 
computer interaction” and “socio-technical systems”. This boundary terri- 
tory is widely occupied by the engineering sciences and their techno-morph 
approaches, such as the ergonomic models of the user as a body machine 
and a sensory mechanism, or the psychological models of “human factors” 
and “adaptive organism”. It seems that the social sciences have given up this 
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terrain at the limits of the social sphere, supposedly because they fear the 
contagious contact with “objectuality”'! and “materiality”'?. Exceptions from 
this theoretical withdrawal can typically be discovered in the cultural media 
and science and technology studies. Bruno Latour has developed the most 
ambitious approach to re-present the things in the polity and to “re-assemble 
the social”, including human and nonhumans.'? In my view, his actor-net- 
work methodology succeeds very well in bringing the “missing masses”" into 
the collective play, but the semiotics of actants!? cultivates a certain blind- 
ness towards observable actions and interactions and underrates processes 
of sense-making. Basing social theory in pragmatism may perhaps help to 
overcome such weaknesses. ° 

Pragmatism’s social theory of interaction has been shown to be fruit- 
ful in explaining the production of social meaning by interpersonal interac- 
tions. This approach can also be used to analyse the relations of interactivity 
between people and physical objects. Georg Herbert Mead is famous for his 
comparative interaction analysis of two dogs fighting with one another and 
of two men boxing and faking against one another.!” He has developed a 
not so well-known, but remarkable piece of theory about human interaction 
with physical objects: Children start to draw distinctions between different 
kinds of objects (own body, outside objects, moving, and living objects), after 
they have learnt the interpersonal role-model of social interaction.'® They 
analyse the activities and attributes of physical objects by taking over the 
role of them, as they have learned it by playing mother’s or sister’s role. 
Being heavy, flexible, moving, having an outer surface and an inner kernel, 
making noises and behaving in an unanticipated way, all these features of 
objects are experienced in children’s play with stationary, mobile and inter- 
active objects. Socialisation encompasses both processes: the interpersonal 
interaction between people, but also the interactivity with physical and sym- 
bolic objects. 

This integrated view on inter-agency has implications for our own enter- 
prise here, namely the inquiry into the changing character of advanced tech- 
nologies and its consequences for human-technology relations. As long as 
technologies, such as simple tools and machines, can be characterised as 
passive or semi-active means, they are used in an instrumental mode: People 
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take and handle them to attain their goals at work or in other everyday life 
situations. The effective action of a tool or a machine is incorporated in its 
design, like the hammer’s long shaft and heavy weight at the end or the 
engine’s encapsulated explosion and the spark generated by the turn of the 
key. Therefore, the user can integrate these objects as mere instruments into 
his action. One can immediately begin using this kind of technology, and one 
can rely on the fixed function and the repetitious operations. One neither 
has to choose options out of a menu of options, nor is one involved in a dia- 
logue with the machine. The only resistance or unexpected re-action of the 
technology would appear if the machine is out of order or the user is com- 
pletely incompetent. When sociologists speak of “instrumental action”, then 
they refer to this kind of unmediated instrumental relation between a man or 
a woman and a machine or a tool. 

Gadgets and machines with higher complexity must be instructed before 
they start their efficient and useful activities. Simple versions of instruction 
can be already found on the classical tool machines: The craftsman instructed 
the machine by turning wheels and tuning measurement instruments. What 
started as a slow specialisation of instructing machines by Jacquard cards, 
paper stripes or record play-back was revolutionised by the invention of 
computer control and software programming from the 1940s onwards. The 
instruction of the machines’ activities became a separate domain. In the long 
run, the devices were miniaturised and integrated into nearly every machine 
and gadget. They were turned into programmable machines, interactive media 
or smart objects. The instrumental use was changed: It was the beginning of 
an instructive-communicative relation between people and objects. 

The rise of a third kind of relation can be observed when the machine asks 
back: Can I help you? Do you really want to delete the document? Please, tell 
me what makes you so sad? What looks like a dialogue between a woman 
at the reception and a guest, an assistant and his boss or a doctor and a 
client was the beginning of a new kind of relation between people talking 
to the machine on the one hand, and software programs that took over the 
roles of communicators, coordinators and agents of all kinds on the other 
hand. Weizenbaum’s ELIZA program showed only marginal changes on the 
program’s side.'® The program’s reaction was restricted to take up some key 
words of the client’s answer and to integrate them in a set of pre-given ques- 
tion sentences. Nowadays, the software agent technology has developed a 
much wider range of capacities to show higher levels of agency. The agents 
can deviate from the standard expectations. They can choose an activity out 
of a bundle of activities. They can assimilate their behaviour to the personal 
user. They can normalise their behaviour by drawing from statistics, and 
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they can change their behaviour by case-based reasoning. These features 
of agency force the user to conceive the relation as if an intelligent agent or 
partner were acting on the other side. Under these conditions of contingency 
and interagency, interactive-communicative relations are emerging. 

Human-technology relations change when technologies are turned into 
more active agents and agencies. The instrumental relation that is typical 
for using tools in craft work and using machines like a tool is fading or only 
stage-managed as an illusion. The push on the button, the foot on the brake, 
and the click with the mouse trigger the activities between several agencies 
that more or less guide the machine, delegate the information-finding to 
Google’s search algorithms, or confront the user with unexpected offerings 
and assistance because the profiling programs have made the user into an 
object. The user of this type of advanced technologies is neither the master of 
the machine nor the slave of the technological system, neither the sovereign 
of his action nor the victim of media’s manipulation. A different concept is 
needed to decide the question of mastery or manipulation, case by case. The 
wider concept of inter-agency replaces the narrow one ofinstrumental use and 
of the perversion of means and goals. The more precisely both activities, the 
agency of objects and the inter-agency between objects, can be observed, the 
more the human-technology relation shows features of complex and contin- 
gent interactivity. Then the instrumental relation is only one particular case 
of an interrelationship. Relations of instructive and communicative interac- 
tivity are the other cases. They will dominate in the future, because nearly all 
kinds of technical objects will be equipped with programmed agency and will 
be made able to communicate with their environment.?° 


4. A gradual model of agency: 
Analysing humans, machines, and programs in action 


The level of human agency is not necessarily always higher than the 
agency of machines and programs. Now we will bring together the two lines of 
argumentation that have been presented before separately. When people are 
in action, their level of agency is not always the highest possible one. They 
may act routinely, like handing over five 100 Euro notes at the bank counter. 
Or they may even do something without any intention, because they follow 
a hidden curriculum of a repressed desire. Reflexive action takes place when 
problems arise or irritations emerge in the course of action. Then people can 
switch from subconscious or routine action over to the next higher level of 
agency, searching for alternative courses of action or reflecting on the moral 
meaning. If one were to count the activities of people, only five percent could 
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be classified as actions with reflected intentions. The rest follow practical 
reasons that could be mostly explained if asked for, or they follow everyday 
routines that often lack even practical reasons.?! 

When machines and programs are in action, their level of agency can 
be higher than usually perceived. Cash machines hand over the money like 
the human actor, at the same time examining the client’s identity and credit 
line, varying the number of notes, signaling misuse, and stopping its activi- 
ties. Even more, video surveillance cameras can be combined with pattern 
recognition software, interactive data-banks, and programs that process and 
mail notices of payment dues. They execute and coordinate actions a lot of 
police men on the street and employees in the offices would be needed for. 
Very simple dispositions are inscribed in this really existing London City law 
enforcement system. One can imagine multi-agent-systems to assist space 
flights or financial brokering whose software agents are equipped with even 
higher ranges of belief, desire and intention capacities in order to learn from 
reactions of other agents and from changing environments. 

When the fundamental duality is to be be overcome of giving all of the 
action to the people and no parts of the action to the objects, then a concept 
of agency is required which also works with lower qualifications of the case 
of what an action is, on the one side. At the other side, it has to be more 
sophisticated about the question of what kind of action do we observe. Thus 
a gradual, three-level model of agency was developed, thereby referring to 
and distancing oneself from Giddens’ three-level model of action and Latour’s 
flattened concept of agency.” Giddens distinguishes three levels of an action: 
a first one where a difference of state is produced, a second one where a 
difference of options is possible, and a third one where actors can give an 
explanation for their action if asked.?? We do not understand these levels as 
a necessary condition of action, but we interpret them as different levels of 
agency. We call these three levels “causality”, “contingency”, and “intentional- 
ity”. Latour, however, pleads for a methodological and ontological symmetry 
and reduces all action to his flattened concept of agency.” We share his anti- 
dualistic methodology, but we insist on levels and degrees of agency. 

On the first level of causality, we start with a weak term of action. Agency 
of this kind means an efficient behaviour, a behaviour that exerts influence 
or has effects, as in the Latin term “agere” or in Latour’s term “actant” or 
Callon’s term “translation”. Under the performative aspect on this level, it 
doesn’t make any difference whether humans, machines or programs execute 
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the action. The money is handed over either by cash machines or by bank 
employees. The situation changes when greater irritations and more options 
come into play. 

On the second level of contingency, the criterion of contingent action is 
required, which means the capacity to act in a different way and to choose 
between options. When the environment changes, the routine action pro- 
gram has to be changed and adapted to it, by people as well as by programs. 
Another possibility arises, when one’s own action program is changed in such 
a way that its consequences are not immediately transparent and account- 
able for the others. When technologies reach this level of contingency, they 
cannot be used as immediate instruments any longer, and do not follow the 
paradigm of command and execution, as has been demonstrated in the previ- 
ous chapter. Instrumentality is replaced by relations of interactivity. Dialogical 
inter-faces and internal user-modeling increase the action level. Interactive 
videogames create spaces of high virtual contingency” that simulate human 
user’s action. These technologies function like a Turing test:?” they make it 
nearly impossible to discriminate between human-enacted and computer- 
enacted characters in the play. 

On the third level of intentionality, the species of reflexive and intentional 
action is allocated. As long as intentionality is by definition ascribed to con- 
scious and knowledgeable human actors only, this level is the domain of 
meaningful action that is oriented to the supposed meaningful action of other 
actors. Chessplaying programs cannot literally have the intention to win a 
game, but they can be constructed as if they had an intentional structure — 
the philosopher Dennett calls this “from an intentional stance””®. Software 
agents cannot cooperate with others in a bodily manner and trust them 
under the explicit belief of augmenting their chances to reach a common goal. 
However, they can be equipped with an intentional vocabulary by which they 
really coordinate and communicate their activities as human actors do, with 
similar semantics. On this level, we plead against a substantial definition of 
action that excludes inquiries into agency. Instead we follow pragmatism, 
which means following all kinds of agencies and focussing on the observable 
practices in which cases the vocabulary of intentionality is used for the con- 
trol or interpretation of activities of people as well as of technical objects.° 
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LEVELS low >>> DEGREES >>> high 
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II. >>> up to guidance by complex semantics 
Intentionality: >>> from ascription of simple dispositions 

I. >>> up to self-generation of actions 
Contingency: >>> from selection of pre-selected options 

I. >>> up to permanent re-structuring of action 
Causality: >>> from short-time irritation 


Fig. 3. Levels and grades of agency? 

This gradual and multi-level model of agency gives us the possibility to 
escape the dilemma of having to either reserve agency up to the humans 
or to flatten the concept of agency unnecessarily. Neither are we forced to 
claim that the activities of humans, machines and programs are substantially 
the same kind of behaviour. Nor do we have to stick to the conception that 
human action and technical operation are fundamentally different from one 
another. This gradual concept of agency opens up a wide range of possibilities 
to identify and to classify kinds and intensities of agency without regards to 
the substantial character of the unit that is in action. Thus the question of 
where the action is can be transformed into an empirical question. 


5. Distributed agency: The very idea 


The question of where the action is cannot be answered unless the answer 
to a second question has been clearly decided: What is the adequate unit of 
action? Conventionally, we suppose a single human actor to be the adequate 
unit of action: the philosopher who thinks and ergo knows that he exists, the 
employee who hands over the bank notes, the pilot who flies two hundred 
tourists to Tenerife airport, and so on. But let us look more precisely at the 
streams of actions from which an action arises. It arises as a distinct action, 
because it is sectioned off, retrospectively emphasised, and ascribed to a sin- 
gle unit, an actor or an author. 

We would have never heard of Descartes’ thought act if he had not writ- 
ten down his famous sentence with a pencil on paper. Even more, the work- 
ing actions of dozens of printers were needed to distribute the phrase in 
hard-covered editions. Additionally, the teaching of hundreds of philosophy 
professors was necessary to diffuse the message under many thousands of 
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students. Perhaps this thought act never took place as a single action at one 
place. Descartes was connected with a lot of thinkers whose arguments he 
received and whose papers he read. Perhaps the foundational thought act 
that is ascribed to him could have been discovered at many loci in that time, 
as if it would be very much “in the air”.?! The act of writing interrupts this 
continuous chain of acts and turns it into the unique philosophical thought 
action that changed the world or at least the world view. The act of writing 
the sentence down by one single actor is emphasised, but both, the flux of 
thought acts before and the sequences of actions afterwards, such as print- 
ing, distributing, reading, teaching and learning, were put into brackets and 
neglected. It is an efficient strategy of teaching and tradition-building to 
attribute a thought act to one author because it reduces cognitive and social 
complexity. However, if we are doing research and inquire into the places, 
faces and activities where the action really is, we should follow all possible 
actors and agencies to the many loci of agency.*” 


5.1 Distributed agency I: 
From a single actor to many loci of agency 


A thought experiment will be used to introduce the second part of the 
paper: Let us answer precisely the question: Who or what is acting in the case 
of flying the tourists to Tenerife? 


PEOPLE MACHINES PROGRAMS 

Pilot? Jet engine? Auto-pilot software? 
Co-pilot? Elevator, Rudder? Navigation card and system? 
Radio operator? Radio equipment? Radio signals and codes? 
Flight-controller? Radar unit? Radar screening? 

Tourist office? Booking machine? Reservation software? 

Airline company? Aviation technology? Technological R&D plans? 
Aviation industry? Air traffic system? Roadmaps for infrastructure? 


Fig. 4. List of actors and agencies in the flight case 
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Humanists and social scientists focus on the people’s side in the list. Their 
first and most plausible answer will be that the human pilot is the acting unit 
that flies the tourists to Tenerife. He is conscious of the goal, the methods and 
instruments. He reflects on possible interventions into the path of the aircraft 
and deviations. Finally, he can be made responsible for the flight because 
he has the power of command and control. But a first uncertainty appears 
when one is confronted with the question: Doesn’t the captain have at least 
one radio-operator at his side? We know from some cases of accidents that 
the communicative actions between pilot and co-pilot or between pilot and 
flight-controller have been critical for the flight action: the consequence can 
entail escaping a collision or not. So we learn that agency can be divided 
between several human actors. The acting unit, then, is either the team on 
board or the locally dispersed assembly of people on board and at several 
control centres on earth. A further question raises other doubts about the 
single heroic actor: Does the captain or this group of navigators and controller 
really plan the flight action? No, it was the air line which planned the route, 
the time and the final departure. It needs more than 200 paying passengers 
so that the action can take place. In comparison to this powerful principal 
agent the other actors fall back in the role of executing agents. The company 
is the so-called collective actor which plans, decides, and controls the flight 
action to Tenerife. In sum, four different units of action can be distinguished 
on the people’s side: a single human actor, a social group or team, a dispersed 
association of people being in interaction by a division of work, and a collec- 
tive actor that coordinates activities towards a goal. Certainly, human agency 
is multiplied, divided, distributed, and connected. 

Encouraged by our gradual concept of agency, one may dare to insist 
on a more precise answer to the question of what actors and other agen- 
cies contribute to the flight action. Engineers and scientists probably would 
emphasise the role of machines and programs. Their first and most plausible 
answer would be: No pilot and no flight without up-currents or artificial driv- 
ers, like propellers or jet propulsion! Elevators and rudders give the air plane 
the direction, and the radio and radar equipment enables the plane to find its 
position and to correct its route. As we have discussed earlier, the agency on 
this low level of causality doesn’t really add new explanatory power. But the 
situation changes completely when these machine technologies and commu- 
nication media are in close intra-activity with the agencies that are enlisted 
on the programs’ side. For the most time of the flight, the flight action and 
the many sub-actions are delegated to the auto-pilot. This is a combination 
of many different software programs that are continually measuring, monitor- 
ing, and computing, but also actively correcting the height, tempo and direc- 
tion of the flight. The automatic landing system sometimes even restricts the 
human pilot from intervening into the action. In sum, the unit of technical 
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agency is constantly changing and growing towards a highly combinatory and 
relatively autonomous technological system. It starts with wings and rud- 
ders. It develops into an aggregated technological system integrating many 
sub-units such as propulsion, navigation, and communication systems. A 
qualitative shift in the level of agency is achieved at the end, when advanced 
computer programs take over the planning, control and navigation activities, 
especially their intelligent coordination, and even more when the flying plane 
itself is turned into one agent in a more extended and self-regulating air traf- 
fic system. 

In the end, we see that it is not so easy to define a human and, in particu- 
lar, a social action. Philosophical and sociological textbooks may help to think 
about the criteria. The authors usually start with a concept of action that is 
isolated from the stream of other actions and that is idealised in a certain 
way. The “ego” is the unit that creates changes, and chooses and defines the 
situation, like God the creator. One can call this concept of action “agency ex 
nihilo” and contrast it with an alternative one, “agency in medias res”**, that 
reconstructs action out of the many activities before and around the focused 
action. Flying 200 tourists to Tenerife is not the instrumental action of a pilot 
navigating the plane to Tenerife airport. It is one activity that is combined 
with other activities of controlling and communication. It is additionally inte- 
grated in the commercial activities of an airline company. Finally, it is also 
nested in the activities of a highly complex organised system within air traf- 
fic, the aviation sector and the tourist industry. Looking at the activities from 
this perspective, one discovers many loci of agency instead of one single actor. 
One can reconstruct the flight action as the commercial action of a collective 
actor or even a network of organisations** which hire people, invest in new 
planes, lobby for public support, advertise cheap charter flights, and organise 
the flight route. 

Looking at the technical side of the list, the talk of gadgets and machines 
as simple means of action underrates both the complexity of aggregated tech- 
nical systems and the self-activeness of programmed and nested systems. 
The collection of many devices and the compilation of different types of tech- 
nologies cannot be handled like bigger tool-boxes with an increasing number 
of instruments in it. These interrelated parts build highly complex systems, 
with many planned intra-activities and some unforeseen interferences,’ so 
that they lose the clear transparency of an instrument and require strat- 
egies of interactivity for their control. The combination of nearly all parts 
with computing and communication capacities converts them into pro-active 
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agents that often are connected in relatively autonomous systems on a higher 
level, like the automatic landing system or the internet-based reservation 
and booking system. As the advanced technologies mostly simulate human 
actions, the different tasks, roles and competencies and actually also the 
social mechanisms of coordination, it makes sense to describe these activities 
and intra-activities with the vocabulary of action and inter-agency. It is the 
adequate way to discover the many loci of technological agency. 


5.2 Distributed agency II: 
From homogeneous agency to hybrid constellations 


In the predominant dualist tradition of thought, the social and cultural 
world of human action, and the material and artificial world of technological 
operation are separated from one another. On the one hand, social scientists 
focus on the motives and expectations of people, such as pilots and flight 
ticket sellers, and on the kinds of social organisation. They reconstruct a 
homogeneous world of symbolic interaction and communication purified of 
physical objects. On the other hand, engineering scientists are preoccupied 
with questions of setting something going, such as air planes or software pro- 
grams, and of improving the effectiveness or safety of technological configu- 
rations. They construct a homogenous world of forced movements and func- 
tioning technological systems purified from social interests and human users. 
Facing the growing interrelatedness of problems of nature and of society, such 
as man-made climate change or artificial stem cell growing, and facing the 
co-construction of socio-technical systems made of people, machines and 
programs, one may, however, ask whether a non-dualist conceptual approach 
could help to make these hybrid constellations a sound subject of research. 

At the borders between the two academic cultures, we already observe 
regular border traffic and even conceptual bridge-building. From research 
in technology and organisations, approaches are being pushed forward that 
respond to the strong interdependency between the material and the social, 
such as the Tavistock approach of socio-technical systems,*° the concept 
of large technical systems consisting of people, organisations, material and 
symbolic artifacts,” and comparative analysis of high risk systems screening 
them along aspects of complexity and interaction between human and non- 
human elements.** The most influential approaches took research in science 
and technology as their point of departure. Some researchers of this area 
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argue against bridging and proposed a radical change of perspective,” such 
as particularly the adherents of the ANT approach, but also of the concepts 
of “objectuality’*°, “socio-technical agency”, and of “material agency”*. 
Research in media and culture is actually a growing third branch where hybrid 
constellations are the new subjects, like being a “cyborg”, “technoscience” or 
living in “virtual life”.* Bridge-building and trans-disciplinary concept-devel- 
opment can be also observed at the science side of the border. Particularly the 
engineering sciences cross the border and take up concepts of the humanities 
and the social sciences. Interface designers integrate psychological concepts 
of cognition and sociological concepts of routine-building and role speciali- 
sation. Designers of software agents apply philosophical concepts of mind, 
belief and intention. And the architectural designers of multi-agent systems 
use sociological concepts of trust, contractual, and market relations. 

From the dualist point of view it makes sense to keep the two territo- 
ries separated. A lot of arguments can be mobilised for this decision, such 
as the ontological differences between people and machines,** the episte- 
mological differences between the disciplines, the institutional differences 
between social organisations and technical configurations and so on. But 
these differences lose their relevance under certain conditions: When human 
actions, machine operations and programmed activities are so closely knit 
together that they form a “seamless web”*, then it makes sense to analyse 
this hybrid constellation as a heterogeneous network of activities and inter- 
activities. When a human action such as flying an Airbus or searching for a 
certain piece of information in hundreds of libraries, millions of books, and 
trillions of files can only be executed with the assistance and intervention of 
hundreds of other agencies, then it is urgent to develop a concept of agency 
that acknowledges all these agencies, though they are heterogeneous in sub- 
stance. And finally, when programmed machine operation is developed such 
that it should execute delegated actions under conditions of contingency, and 
when it is implemented in open systems that are constructed by the interac- 
tions between the software agents, then one should integrate these agencies 
into the framework of analysis. Therefore a concept of distributed agency is 
argued for here only under these conditions of advanced technologies and 
instituted hybrid constellations. 
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Coming back to our flight action example, the answer to the question of 
what the adequate unit of action is can now be given: It is the hybrid con- 
stellation of people, machines, and programs. It is the mode in which the 
agencies of the heterogeneous instances are distributed and connected with 
one another and the level of agency that is given to them in certain situa- 
tions. It is neither the single or the collective human actor, nor the technical 
artifact alone, nor the combined technical system. It is the mixed ensemble 
made of all elements on both sides of the border. One can call it a collective 
agency, alluding to the term “collective actor”. This collective is constituted 
by the distributed activities of heterogeneous units in comparison to what is 
referred to by the other term, which is built out of the homogeneous stuff of 
human actions. 


5.3 Distributed Agency III: 
From hierarchy to framed interactivity 


Two modes can be distinguished in which actions can be divided and 
integrated: a hierarchical mode in which specialised activities are strongly 
integrated, and an interactive mode in which distributed modal units are 
weakly coupled. In the sociology of organising, they are often referred to when 
distinctions are made between bureaucratic and organic models or between 
strongly or weakly coupled systems.*® Observing complex organisations, one 
learns very quickly that hierarchical integration is only the most effective 
mode for divisions with fixed inputs, routine processing and stable environ- 
ment, like the mass-production of things. Units that are confronted with 
changing inputs, many variations in the process and dynamic environments 
require a more interactive, flexible and open mode of organising, like R&D 
departments or creative industries. Most of the modern organisations show a 
mix of both modes of integration, mechanising the routine parts and learning 
by interactivity with the environment. 

It was taken for granted up to now that the hierarchical mode was the 
only and the best way to specialise and integrate technologies. It was the 
paradigm for the first machine made of the forced movements of people work- 
ing based upon a division of labour to build the pyramids, and also for the 
ongoing process we call mechanisation.?” Technologies are defined by their 
capacity to force different activities into a mechanical form that is reliable, 
accountable and usable as a mean to solve particular problems in an effec- 
tive and expected way.*® Tasks are divided between many specialised parts 
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and integrated by linear chains of operations and hierarchical schemes of 
processing. However, this dominant mode and its supposed universality are 
now being challenged. Some technical configurations and socio-technical 
constellations can be observed that are integrated in a different mode that 
resembles the above mentioned interactive mode. 

One can already observe small deviations from the strong mechanical 
mode when looking at the feed-back loops of cybernetic systems. The sand- 
wich architecture of the computer also shows a loosening of point-to-point 
determination between its physical machine level and the logical level and 
the program language level. A further milestone on the path of breaking the 
linearity was the concept of “distributed computing””. It started with the sim- 
ple problem of distributing computing time, but gained its momentum when 
a new generation of software programs were developed that used fuzzy logic, 
distributed artificial intelligence, agent-oriented programming, and models 
of socionics in order to admit distributed activities and parallel processes. 
Particularly in social computing’! and in socionics°?, many modes of interac- 
tive integration were developed that were in opposition to the hierarchical 
mode. 

Another milestone was the development of the concept of “distributed 
cognition”. The psychologist Hutchins criticises the dominant model of 
individual problem-solving in the cognitive sciences that supplies the artifi- 
cial intelligence community with a construction plan. It presupposes sepa- 
rated and functional specialised activities that can be easily aggregated. Also 
being an ethnographer, Hutchins observed the techniques of navigation “in 
the wild” and “in medias res”: He studied precisely how the Polynesian long- 
distance sailors performed navigation in the wide Pacific ocean though they 
had no sophisticated nautical instruments, and how a navigation team on a 
warship maneuvered their long ship into the small harbor entrance of San 
Diego though its nautical system was damaged. In this way he discovered a 
mode of self-organised integration between distributed processes of cognitive 
activities. The cognitive action of positioning was organised as a distributed 
process that was performed by some people with different practices, natural 
objects and technical instruments. The critical point for our argumentation 
here is his observation that these distributed processes did not require any 
planning, functional specialisation or hierarchical integration. Their mode of 
integration was described as a natural process of loose coupling, overlapping 
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activities, experimental adaptation, and a step-by-step stabilisation of a com- 
mon frame for the interactions.°* 

The concept of “distributed agency” that is presented in this paper follows 
the lines that were started by those concepts of “distributed computing” and 
“distributed cognition”. The first step towards constructing this concept of 
distributed agency has been to demonstrate that human action is distrib- 
uted between many loci and instances that plan, control, and execute the 
activities. Distributed action means that someone searches for significant 
marks, someone else measures the angles, a third person plots by drawing a 
line, and others count, communicate and correct the data. All these interac- 
tions between them constitute an observable unit of action called navigation. 
This kind of distribution can also be transferred to computer operations. The 
action of sending a message to a certain person can be broken down into 
many activities at different places, such as encoding, packaging, addressing, 
transporting, and reading TCP/IP protocols at the PC, at the server, at the 
local area network, or at one of the knots of the worldwide web. 

The second step has been to cross the Rubicon between the two homog- 
enous spheres of human action and technological operation: distributed 
agency then refers to hybrid constellations made of heterogeneous units of 
agency. Moving objects such as the sun and the currents of water, measuring 
instruments, counting tables, and carved records participated in the action of 
navigation. As we have argued before, objects participate more activelyand on 
a higher level of agency when the nautical pilot program and the automatic 
navigation system are in action and in close intra-activity with one another. 

The third step now emphasises the two modes of integration. They dif- 
fer in how the units are divided, how they are processed, and how they are 
connected with one another. The dominant hierarchical mode of integration 
prolongs the traditional line that allows us to treat even complex technologies 
and hybrid constellations as reliable means and robust mechanisms. The 
mode of framed interactivity is rarely implemented because it deviates from 
the well-known and trusted master-slave relation. The technological units are 
given more freedom of choice and higher levels of agency in order to enrich 
their capacity of assistance and to strengthen their role as relatively autono- 
mous agents. 
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MODES: HIERARCHY FRAMED INTERACTIVITY 
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Type of Division of work Distributed activities 

Differentiation Functional specialisation Fragmented units 

Type of Mechanical Organic 

Organisation Bureaucratic Open System 

Type of Linear sequences Parallel processes 

Connection Strongly coupled Loosely coupled 
Fixed and general rules Flexible, situated, and specific rules 
Pre-Programmed Framed Self-adaptation 
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Though the mode of framed interactivity has rarely been implemented 
up to now, this mode may become a new paradigm for the design of future 
constellations. It is currently sought after in many different places: in labo- 
ratories of distributed intelligence, in research and development clusters on 
robotics, man-machine interfaces and new media design, as well as in the 
studios of interactive artists, in the media labs of the entertainment industry, 
and at the software benches of videogame developers. This mode of framed 
interactivity will get its chance to be diffused when the next generation of 
technologies is consciously designed and implemented from the perspective 
of distributed agency, when the frames of heterogeneous agencies are bal- 
anced and tuned to each other, and when a new generation of users is coming 
up that is used to the new experiences with interactivity. 


86 


References 


Adelmann, Robert/Langheinrich, Marc/Floerkemeier, Christian (2006): »Toolkit for 
Bar Code Recognition and Resolving on Camera Phones - Jump Starting the Internet 
of Things«. In: Christian Hochberger/Rüdiger Liskowsky (Eds.), Informatik 2006 - 
Informatik für Menschen, Band 2, Beiträge der 36. Jahrestagung der Gesellschaft 
für Informatik e.V. (GI), 2.-6. Oktober 2006 in Dresden, 366-373. 

Akrich, Madeline/Latour, Bruno (1992): »A Summary of a Convenient Vocabulary for 
the Semiotics of Humans and Nonhuman Assemblies«. In: Wiebe E. Bijker/ John Law 
(Eds.), Shaping Technology - Building Society, Cambridge: MIT Press, 259-264. 

Callon, Michel (1986): »Some Elements of a Sociology of Translation: Domestication of 
the Scallops and the Fishermen of St. Brieux Bay«. In: John Law (Ed.), Power, Action, 
and Belief, London: Routledge, 196-229. 

Callon, Michel/Latour, Bruno (1992): »Do not throw out the baby with the Bath school! 
A reply to Collins and Yearley«. In: Andrew Pickering (Ed.), Science as Practice and 
Culture, Chicago: University of Chicago Press, 343-368. 

Coleman, John S. (1990): Foundations of Social Theory, Cambridge: The Belknap Press 
of Harvard University Press. 

Collins, Harry M./Yearley, Steven (1992): »Epistemological Chicken«. In: Andrew 
Pickering (Ed.), Science as Practice and Culture, Chicago: University of Chicago 
Press, 301-326. 

Collins, Harry M./Kusch, Martin (1998): The Shape of Actions. What Humans and 
Machines Can Do, Cambridge: MIT Press. 

Dennett, Daniel (1987): The Intentional Stance, Cambridge, MA: MIT Press. 

Esposito, Elena (1995): »Illusion und Virtualitat: Kommunikative Veranderungen der 
Fiktion«. In: Werner Rammert (Ed.), Soziologie und künstliche Intelligenz. Produkte 
und Probleme einer Hochtechnologie, Frankfurt a.M./New York: Campus, 187-216. 

Foerster, Heinz von (1985): Sicht und Einsicht. Versuche zu einer operativen 
Erkenntnistheorie, Braunschweig: Vieweg. 

Fuller, Steve (1994): »Making Agency Count. A Brief Foray into the Foundation of Social 
Theory«. American Behavioral Scientist 37, 741-753. 

Giddens, Anthony (1984): The Constitution of Society: Outline of the Theory of 
Structuration, Cambridge: Polity Press. 

Giedion, Siegfried (1948): Mechanization takes command: a contribution to anonymous 
history, New York: Oxford University Press. 

Girard, Monique/Stark, David (2007): »Socio-technologies of Assembly: Sense-Making 
and Demonstration in Rebuilding Lower Manhattan«. In: David Lazar /Viktor Mayer- 
Schoenberger (Eds.), Governance and Information: The Rewiring of Governance and 
Deliberation in the 21st Century, Oxford: Oxford University Press, 145-176. 

Goffman, Erving (1969): Strategic Interaction, New York: Routledge. 


Haraway, Donna J. (1991): Simians, Cyborgs, and Women: The Reinvention of Nature, 


New York: Routledge. 


Haraway, Donna J. (1997): Modest_Witness@Second_ Millennium. FemaleMan_Meets_ 
OncoMouse. Feminism and Technoscience, New York: Routledge. 

Hewitt, Carl E. (1977) »Viewing Control Structures as Patterns of Passing Messages«. 
Artificial Intelligence 8, 323-364. 

Hughes, Thomas P (1986): »The Seamless Web: Technology, Science, Etcetera, Etcetera«. 
Social Studies 16, 281-292. 

Hughes, Thomas P (1987): »The Evolution of Large Technological Systems«. In: Wiebe 
E. Bijker/Thomas P. Hughes/Trevor. J. Pinch (Eds.), The Social Construction of 
Technological Systems, Cambridge: MIT Press, 51-82. 

Hughes, Thomas P. (1998): »Learning to Navigate. Understanding Practice. Perspectives 
on Activity and Context«. In: Seth Chaiklin/Jean Lave (Eds.), Unterstanding 
Practice. Perspectives on Activity and Context, Cambridge: Cambridge University 
Press, 35-63. 

Hutchins, Edwin (1996): Cognition in the Wild, Cambridge: MIT Press. 

Ihde, Don/Seliger, Evan (2003): Chasing Technoscience: Matrix for Materiality, 
Bloomington: Indiana University Press. 

Joas, Hans (1989): Praktische Intersubjektivitat. Die Entwicklung des Werkes von G. H. 
Mead, Frankfurt a.M.: Suhrkamp. 

Kaufmann, Jean-Claude (2008): Was sich liebt, das nervt sich, Konstanz: UVK-Verlag. 

Knorr Cetina, Karin (1998): »Sozialitat mit Objekten. Soziale Beziehungen in post- 
traditionalen Wissensgesellschaften«. In: Werner Rammert (Ed.), Technik und 
Sozialtheorie, Frankfurt a.M.: Campus, 83-120. 

Latour, Bruno (1987): Science in Action: How to Follow Scientists and Engineers 
Through Society, Cambridge: Harvard University Press. 

Latour, Bruno (1988): »Mixing Humans and Nonhumans together: The Sociology of a 
Door-Closer«. Social Problems 35, 298-310. 

Latour, Bruno (1992): »Where are the Missing Masses? The Sociology of aFew Mundane 
Artifacts«. In: Wiebe E. Bijker/John Law (Eds.), Shaping Technology — Building 
Society, Cambridge: MIT Press, 225-258. 

Latour, Bruno (1994): »On Technical Mediation: The Messenger Lectures on the Evolution 
of Civilization«. Common Knowledge 3, 29-64. 

Latour, Bruno (1996): »On Interobjectivity«. Mind, Culture, and Activity 3, 228-245. 

Latour, Bruno (2005): Reassembling the Social. An Introduction to Actor-Network- 
Theory, Oxford: Oxford University Press. 

Leroi-Gourhan, Andre (1980): Hand und Wort. Die Evolution von Technik, Sprache und 
Kunst, Frankfurt a.M.: Suhrkamp. 

Malsch, Thomas (2001): »Naming the Unnamable: Socionics or the Sociological Turn of/ 
to Distributed Artificial Intelligence«. Autonomous Agents and Multi-Agent Systems 
4, 155-186. 

Mayntz, Renate/Schneider, Volker (1988): »The Dynamics of System Development in 


Comparative Perspective: Interactive Videotext in Germany, France, and Britain«. 


88 


In: Renate Mayntz/Thomas P Hughes (Eds.), The Development of Large Technical 
Systems, Bolder: Westview Press, 263-298. 

McCarthy, E. Doyle (1984): »Toward a Sociology of the Physical World: Mead on Physical 
Objects«. Studies in Symbolic Interactionism 8, 105-121. 

Mead, George H. (1932): The Philosophy of the Present, Edited by Arthur E. Murphy, La 
Salle: Open Court. 

Mead, George H. (1963): Mind, self and society, Chicago: University of Chicago Press. 

Meister, Martin/Schröter, Kay/Urbig, Diemo/Lettkemann, Eric/Burkhard, Hans- 
Dieter/Rammert, Werner (2007): »Construction and Evaluation of Social Agents in 
Hybrid Settings: Approach and Experimental Results of the INKA Project«. Journal of 
Artificial Societies and Social Simulation 10 [electronic journal], <http: / /jasss.soc. 
surrey.ac.uk/JASSS.html> (last access: July 2008). 

Merton, Robert K. (1957): »Priorities in Scientific Discovery: A Chapter in the Sociology 
of Science«. American Sociological Review 22, 635-659. 

Perrow, Charles (1984): Normal Accidents: Living With High Risk Technologies, 
Princeton: Princeton University Press. 

Perrow, Charles (1986): Complex Organizations. A Critical Essay, New York: Random 
House. 

Pickering, Andrew (1993): »The Mangle of Practice: Agency and Emergence in the 
Sociology of Science«. American Journal of Sociology 99, 559-589. 

Pickering, Andrew (1995): The Mangle of Practice: Time, Agency and Science, Chicago: 
University of Chicago Press. 

Preda, Alex (2006): »Socio-technical Agency in Financial Markets: The Case of the Stock 
Ticker«. Social Studies of Science 26, 753-782. 

Rammert, Werner (1992): »From Mechanical Engineering to Information Engineering: 
Phenomenology and the Social Roots of an Emerging Type of Technology«. In: Meinolf 
Dierkes/Ute Hoffmann (Eds.), New Technology at the Outset. Social Forces in the 
Shaping of Technological Innovations, Frankfurt a.M.: Campus/Bolder: Westview, 
193-205. 

Rammert, Werner (1998): »Giddens und die Gesellschaft der Heinzelmannchen. Zur 
Soziologie technischer Agenten und Systeme Verteilter Künstlicher Intelligenz«. In: 
Thomas Malsch (Ed.), Sozionik. Soziologische Ansichten über künstliche Sozialität, 
Berlin: Sigma, 91-128. 

Rammert, Werner (2001): »Relations that Constitute Technology and Media that Make 
a Difference: Toward a Social Pragmatic Theory of Technicization«. In: Hans Lenk / 
Matthias Maring (Eds.), Advances and Problems in the Philosophy of Technology, 
Münster: Lit Verlag, 271-290. 

Rammert, Werner (2007): Technik - Handeln - Wissen. Zu einer pragmatistischen 
Technik- und Sozialtheorie, Wiesbaden: VS Verlag. 

Rammert, Werner/Schulz-Schaeffer, Ingo (2002): »Technik und Handeln - Wenn 


soziales Handeln sich auf menschliches Verhalten und technische Artefakte verteilt«. 


89 


In: Werner Rammert/Ingo Schulz-Schaeffer (Eds.), Können Maschinen handeln? 
Frankfurt a.M.: Campus, 11-64. 

Rammert, Werner/Schubert, Cornelius (2006): Technografie. Zur Mikrosoziologie der 
Technik, Frankfurt a.M.: Campus. 

Rumelhart, David/McClelland, James (1986): Parallel Distributed Processing. Vol. I. 
Foundations, Cambridge: MIT Press. 

Schubert, Cornelius (2007): »Risk and safety in the operating theatre. An ethnographic 
study of socio-technical practices«. In: Regula V. Burri/Joseph Dumit (Eds.), 
Biomedicine as culture. Instrumental practices, technoscientific knowledge, and 
new modes of life, London: Routledge, 123-138. 

Schulz-Schaeffer, Ingo (2002): »Innovation durch Konzeptübertragung. Der Rückgriff 
auf Bekanntes bei der Erzeugung technischer Neuerungen am Beispiel der 
Multiagentensystem-Forschung«. Zeitschrift für Soziologie 31, 232-251. 

Schulz-Schaeffer, Ingo (2007): Zugeschriebene Handlungen. Ein Beitrag zur Theorie 
sozialen Handelns, Weilersvist: Velbrück. 

Star, Susan L. (1989): »The Structure of Ill-Structured Solutions: Boundary Objects and 
Heterogeneous Distributed Problem Solving«. In: Michael Huhns/Les Gasser (Eds.), 
Distributed Artificial Intelligence, Menlo Park: Morgan Kauffman, 37-54. 

Teubner, Gunther (2002): »Hybrid Laws. Constitutionalizing Private Governance 
Networks«. In: Robert Kagan/Kenneth Winston (Eds.), Legality and Community, 
Berkeley: California University Press. 

Trist, Eric (1981): The Evolution of Socio-Technical Systems: A conceptual framework 
and an action research program, Toronto: Ontario Quality of Working Life Centre. 

Turing, Alan (1950): »Computing Machinery and Intelligence«. Mind 59, 433-460. 

Turkle, Sherry (1995): Life on the Screen. Identity in the Age of Internet, New York: 
Simon & Schuster. 

Weick, Karl (1976): »Educational Organizations as Loosely Coupled Systems«. 
Administrative Science Quarterly 21, 1-19. 

Weizenbaum, Joseph (1977): Die Macht der Computer und die Ohnmacht der Vernunft, 
Frankfurt a.M.: Suhrkamp. 

Winograd, Terry/Flores, Fernando (1986): Understanding Computers and Cognition. A 
New Foundation for Design, Reading: Addison-Wesley. 

Wooldridge, Michael/Jennings, Nicholas (1995): »Intelligent Agents: Theory and 
Practice«. The Knowledge Engineering Review 10, 115-152. 


90 


Acknowledgements 


I thank Cornelius Schubert for critical comments and Kristina Brandt for tech- 
nical assistance. This paper would not have been written without the DFG-support 
“Technik in Aktion” and the invitation to the ZiF research group “Science in the Context 


of Application” at the University of Bielefeld. 


91 


E ra sS” a 
K Ca 
s : À E r 
p g 
ro $ S P 


ONE CONCEPT 


s S E E s$ S 


$ oS # s 


E s Ka 


Frieder Nake 


o S 


Jürgen Habermas’ essay “Arbeit und Interaktion” first appeared in an 
anthology in 1967. One year later, it was put together with four more of 
his texts and re-published under the title “Technik und Wissenschaft als 
‘Ideologie”.! 

When I first read it in 1972, it was because of the word “labor” in its title. 
I don't remember a thing from the fist reading, and the habitual underlinings 
and marginal remarks on the pages of the book don't indicate from when 
they are. I guess I put the book away on my bookshelf where it remained for 
more than a decade, perhaps untouched. When I read it the second time, 
probably in the 1980s, it was more or less by accident. I was browsing my 
bookshelves in search of something by Marcuse, when out of some intuitive 
reason I grabbed the thin Habermas volume and only then became aware of 
the other word, “interaction”, in the title “labor and interaction”. At the time 
(in the 1980s) I was teaching a four semester cycle in computer graphics 
whose specific topics were foundations, geometric modeling, graphic render- 
ing, and techniques of interaction. Clearly, my bookshelf discovery forced me 
to immediately sit down and study what Habermas had to say on interaction. 
Why was he using the term? 

Imagine the timely context of the situation! Charles P Snow had delivered 
his “The two cultures and the scientific revolution” in Cambridge in 1959.? I 
had then been a student of mathematics at the University of Stuttgart. A year 
before, during a two month internship at IBM in their Böblingen Computing 
Centre, I had had my first encounter with a computer. Max Bense, the pro- 
vocative philosopher at Stuttgart, had surely referred to Snow’s talk and con- 
cept of two cultures. I found myself caught in a great melting pot: studying 
mathematics, the queen of all mental efforts, experiencing the grandeur and 
joy of strict axiomatics, formal concepts, theorems, and proves. From this 
comfortable centre to the left were Bense’s thrilling lectures about aesthetics, 
ontology and, particularly, semiotics; to the right was theoretical and experi- 
mental physics, or the theory of electrical engineering, and more. 

What a time, what a storm! We were a group of friends, trying to under- 
stand what the engineering types told us as well as what came from those in 
the humanities. We felt more and more at home in mathematics and, soon 
enough, in its rather trivial offspring, computing. But now we were confronted 
with Snow’s claim that no communication was possible across the bounda- 
ries between the two types of disciplines we liked so much, because they 
were both exciting in their own way: the scientific and the literary cultures. It 


1 The German original I am working with is Habermas 1969. It contains on pp. 9-46 
the essay “Arbeit und Interaktion. Bemerkungen zu Hegels Jenenser ‘Philosophie des 
Geistes”. It is available in English as “Labor and interaction: comment on Hegel’s Jena 
‘Philosophy of Mind”, which appeared on pp. 142-169 in Habermas 1973. 


2 The text is easily available, e.g. Snow 1993, which contains the original lecture of 
1959 plus The two cultures: a second look, written five years later. I use Kreuzer 1987. 
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must have been puzzling to the young student who in the early morning was 
listening to a great electrical engineer, did his mathematics around noon, a 
bit of programming after lunch, and went to hear about Peircean semiotics in 
the late afternoon, just before rushing away to the opening of some artist’s 
exhibition. 

But now he had to swallow the fact that there was no communication, 
no interaction!? How could that be possible? Wasn't he himself doing exactly 
this: moving back and forth between those areas that - to be sure — had their 
own ways of expressing their findings, certainly, but wasn’t communication 
happening through his own activities, even if restricted to himself and his 
friends? A decade later, in 1968, the great outburst of the youth’s revolution 
against the father’s generation shook the country. First readings about work 
and exploitation and suppression followed — a lot of Marx some years after 
this, and then, a strange sort of revelation. 

Against the largely mathematical background of computer graphics, the 
open and heated discussions about human-computer communication were 
pure excitement. I had published a paper in 1984 on the impossibility that 
humans could ever communicate with computers, if the term “communica- 
tion” was to be taken seriously.* Unix was ruling, Silicon Graphics machines 
were great, but the Apple Macintosh had appeared on the market. We were 
beginning to interact with the computer as never before. The researchers 
at Xerox PARC had done tremendous things that made beautiful surprises 
in the classroom. C. P Snow’s verdict was still present in my thinking. But 
now came the discovery of that second word in the title of Habermas’ essay: 
interaction! 

Reading, on page 9 of this collection of supposedly critical texts by 
Habermas, the word Interaktion not only meant that you were, perhaps, no 
longer restricted to saying “HCI” when you were talking about humans and 
computers and their intricate relations. It meant that you could, perhaps, 
add a totally different perspective than the one usually offered in computing 
circles. So what did I learn, how did I read Habermas? 

In a nutshell, Habermas, in my reading and restricted conclusion, said 
the following. The mediation of subject and object is what constitutes “Mind” 
(Geist). There are three ways how the subject and the object may be related. 
Three categories mediate between subject and object. The three categories are 
language, tool, and family. These terms stand for three patterns of dialectic 
relations, the patterns of symbolic representation, labour process, and recip- 


rocal interaction. 
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3 Nake 1984, pp. 109-118 
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The tool and labour process stand for that mediation of subject and 
object, where the subject changes the state of the object. The tool-relation 
transforms the object into a state better fit for the subject’s needs. 

The language or symbolic representation stands for that mediation of 
subject and object, where the subject observes and describes the state of 
the object. The language relation creates a semiotic layer that stands for the 
object. 

The family or reciprocal interaction stands for that mediation of subject 
and object, where the subject accepts the object as of the same kind and 
capacity as the subject itself. The interaction relation leads to cooperative, 
communicative exchange between subjects that are equal. 

Mind you, this is the simple, naive, and immediate interpretation of a 
probably difficult reflection. The interpretation was by a computer scientist 
who was quite happy to find something in the other faculty which he thought 
could help him. The writing was by a philosopher and social scientist who, by 
the time of his writing, was developing his theory of communicative action, 
which played an important role in his attempt to reconstruct, as he said, 
dialectical materialism. Undoubtedly, dogmatists must have fiercely attacked 
him for working on such a project, whereas more liberal representatives of the 
left may have accepted the premise that Marxism also has the right to evolve 
when social reality changes.* My naive view, however, quite happily suggested 
to me that a systematic approach could be applied in order to introduce an 
interactive mode of using computers. I may have, of course, grossly misun- 
derstood, nevertheless, the threefold distinction may serve a purpose here. 

In the rest of this essay, I will describe three cases of using computers. 
Each case involves two persons in varying positions relative to the machine. 
In each case, we will see the open surface of the computer periphery, which 
will be commented on. We will also see the hidden and more or less inacces- 
sible subface, and will gain some insight into how, and why, the two are nec- 
essarily related to each other. We will discover what the reader may already 
have been aware of all the time: computer things come in pairs. We will briefly 
give a semiotic interpretation of this claim by introducing the concept of an 
algorithmic sign. In conclusion, we will point to the current importance of 
digital media as those media that explore the dialectics of algorithmics and 
aesthetics. 
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Scene 1: Automaton. 
Two persons interacting, waiting for the computer 


The year is 1963 or 1964. Two men are sitting in front of a computer. The 
size of the room may be 15 by 6 meters. No-one else is in there. They are not 
really sitting in front of the computer. The racks and cabinets that make up 
the computer create a system way too large to be sitting in front of. Besides, 
the two are sitting inside a separated space surrounded by glass walls. They 
see the metal cases, lined up in a long row of several meters length, and the 
tape drives next to the cases. Inside the glass room, temperatures are a bit 
more agreeable for humans. Outside, the climate must be closely monitored 
to stay within a small margin to prevent frequent failure of operation. The 
peripherals of the machine share the glass-walled space with the two men; a 
paper tape reader and a tape printer next to a large computer console. 

One of the men has recently gradu- 
ated in mathematics. The other one is 
a researcher in mechanical engineer- 
ing. His theoretical investigations have 
led him to describe the behaviour of 
a metal sheet as a non-linear fourth 
order differential equation with bound- 


ary conditions. The equation can only 


be solved numerically. 
Fig. 1. The operator’s interface. y 


Computer SEL ER56 at Computing 
Centre, University of Stuttgart the computing centre for help, the 
1965. We see dials (lower right) young mathematician was given the 
and push buttons to get at speci- job of cooperating with the senior per- 
fied data. Through the glass 
in the background some of the 
hardware 
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When the researcher approached 


son. At their first meeting the engineer 
explained what he was doing, why he 
was interested in getting the solution, 
even an approximate one, and that he hoped the younger man would do all 
the programming since he himself had no clue of what might be involved 
and could see no chance of changing this. Why should he learn to program? 
He would rather agree on meetings and the admittedly arduous work of 
cooperation. 

A great chance for the young mathematician. He did not take much inter- 
est in the details of the mechanics. For him, it was enough to accept the 
differential equation as his starting point. To which extent it described the 
vibrations of the sheet was the engineer’s responsibility. As a mathemati- 
cian, he was responsible for the selection of a modern numerical method for 
numerical integration (he chose a Fehlberg algorithm). Fine tuning for effi- 
ciency in those old days was an important and creative task. You could come 
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up with improvements week after week. You did the improvements yourself, 
no compiler could take the burden from you (in fact, there was no compiler 
on this machine). But sooner or later the program worked and production 
could begin. 

Since this was a boundary value problem, another difficulty had to be 
tackled: how to satisfy the boundary conditions? Such conditions require 
that a valid solution starts and ends at specified locations. It must leave and 
arrive there under specified directions (making a total of four conditions). 

The rather simple approach was trial-and-error: Start from the prescribed 
position at the left end; solve an “initial-value” problem by arbitrarily assum- 
ing two more initial conditions; compare the calculated endpoint to the right 
with the prescribed goal. If there is a difference (which will, most likely, be 
the case), adjust the arbitrarily chosen initial condition such that, hopefully, 
the discrepancy is diminished, and repeat. This shoot-and-run approach 
will, under not too heavy circumstances and after some systematic attempts, 
come close enough to a realistic solution. 

In those days, computers were incredibly slow when you compare them 
with their performance now, fourty years later. The difference in efficiency 
must be 6 to 10 degrees of magnitude, if not more. The two men, after having 
started the next shot, had to wait for several minutes (up to five), before the 
machine presented the few numbers they needed to judge how close the shot 
had come to the goal. While waiting, they had plenty of time to talk about 
mechanics, mathematics, programs, artificial intelligence, philosophy, poli- 
tics, university gossip, the latest jazz concert, and a dozen more topics. 

The situation thus indicated is heavily interactive! The two intellectuals 
were cooperating to solve a tough problem (they actually spent about two or 
three months before enough data had been generated from the virtual experi- 
ments in the algorithmic laboratory). Enough data was achieved when the 
engineer decided that his theoretical model was now backed up by enough 
empirical evidence. The cooperation between the two men allowed them to 
interact in the most complex and interesting ways. No procedures, no meth- 
ods prevented them from journeying down any of the myriads of alleys open 
to the mind. The two actually became friends. 

Embedded into the human-human interaction were short moments of 
very low-level human-computer interaction. When the result of the last cal- 
culation had become visible, the two discussed it and decided how to pro- 
ceed. Proceeding was defined by a choice of new initial values. The process 
of choosing could have been automated since the goal was well-defined. But 
the amount of extra calculations would then, very likely, increase much more 
rapidly than with personal inspection. Inspection and discussion took advan- 
tage ofthe human capacity to detect patterns, consider context, and be aware 
of the situation and its changes. 
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The machine they were using, an early transistorised decimal (!) computer 
was good enough for the purpose. Its interface displayed current numerical 
values of data stored in memory cells. The two friends, in order to read the 
coordinates of the final destination, had to dial knobs to get at those memory 
locations (given by their absolute addresses), and then lock up the numerical 
contents of that location. 

At the machine level, only a tiny bit of interaction was happening. Really, 
this wasn’t more than looking up and reading some signals — ridiculously low- 
level when compared to what became standard twenty years later. Slow times 
those days, you would say. Two men engaged in watching the computer come 
up with the result of a calculation. They had to wait because the machine was 
used as an automaton. They fed the automaton with data, hit the start button 
for the automaton to do its work, and were thrown back to talking, becoming 
bored, or listening to the radio. 


Scene 2: Tool. One person using 
the computer, the programmer far away 


The year is 2002, nearly forty years later. In terms of technology, the 
world has turned upside down. The technical infrastructure of all processes, 
in the private, economic, administrative, or political realm, is determined by 
data processing. In parts of the world, there is virtually no room that does 
not contain at least one computer. The art historian is sitting at her desk at 
home. Her current field of interest is an area not well known, certainly not 
mainstream, but slowly and steadily gaining interest. Already in her Ph.D. 
thesis a few years ago she started to seriously study the phenomenon that in 
the mid 1960s was called “computer art”. She now generally prefers to use 
the term “digital art”. 

She is preparing for a meeting with her students. Recent work by Manfred 
Mohr will be the topic. He is the German artist who first got access to a com- 
puter controlled drawing machine (often called a “plotter”) in 1969 in Paris. 
He gave up painting in favour of programming, stopped using colour in favour 
of black and white and, later, some grey and silver. He gradually became a 
recognised artist who could make a living from selling his art. He had dis- 
covered his topic in the early 1970s: the cube and its symmetries. In order to 
gain complexity, the cube had become the hypercube of four or five or even 
more dimensions. Like other artists before him — say, e.g., Paul Klee, Josef 


Albers, the concrete artists — he had become a researcher as well. 
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The sensation happened in 2001 when he exhibited large canvasses in 
bright colours.? Our art historian was trying to understand those pictures. 
They had again been exhibited under the title space.color in October 2001 
at the Museum für Konkrete Kunst in Ingolstadt, Germany. Ever since, Mohr 
had been using coloured fields in his paintings (and, later, computer instal- 
lations). He needed six- or eleven-dimensional space and colour to increase 
complexity by orders of magnitude. 

The art historian at her home desk, not being acquainted with geo- 
metric spaces of higher dimensions, tried to understand the algorith- 
mic process that generated some of the pictures in front of her. The algo- 
rithm behind the canvas had been Mohr’s secret for all his productions. 
In his catalogues he had been friendly enough to publish brisk and 
sober definitions of the algorithmic behaviours of the generative proc- 
esses. But now, with colour reappearing in Mohr’s works, she felt lost. 
She had been stud- 
ying the catalogue 
of the Ingolstadt 
exhibition: reading 
statements, analy- 
sing pictures. She 
was unable to grasp 
how the remark 
that something was 
going on in six di- 


mensions could be 
helpful. She turned 
on her laptop com- 


Fig. 2. Left: Manfred Mohr: P-107-f (1999). Right: P-1011-z2 
(2004). Six dimensions behind the left, eleven dimen- 
sions behind the right image 
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puter. Someone had 
given her a program 
called deviceX. It was supposed to help develop some kind of understanding 
of the space.color period of Mohr’s art. 

The name, deviceX, rang two bells in her mind: tools and X-rays. Tools are 
instruments we use to more easily change the state of some material; X-rays 
are dangerous but helpful in looking into the human body. DeviceX could, 
perhaps, be a tool to look into the structure of those paintings. 

Mohr’s paintings of the space.color variety appear - just like any other 
painting does - as a configuration of coloured forms. The configuration of the 
forms corresponds to the geometry of the painting. We may derive from the 
geometry of a painting a more abstract rendition of the same content. This 
abstract rendition may be called the painting’s topology. The abstraction gives 
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5 For the first time, Mohr showed these pictures in June 2001 at Galerie Wack in 


Kaiserslautern, Germany. 


up the particular form of an area, its size and, to a large extent, its location. 
Of the geometry, the topology keeps only one relation: that of neighborhood. 
If two areas are neighbors in the geometry, they must also be neighbors in 
the topology. So if they have a common edge in geometry, their topological 
neighbor-relation is an edge-neighborhood. 

Topology is abstract. It gets described in formal symbolism. We can, how- 
ever, use a minimal visualisation by using squares as the only form features. 
DeviceX makes use of this. The art historian started the program. Soon she 
found out how to use it. Among some other features not of prime interest 
here, the most prominent one is the following (Fig. 3). There is a small rep- 
lica of one of the possible Mohr paintings. It can be grabbed with the mouse 
device and shifted horizontally, left and right. As it is moved further to the 
right, the intricate geometric forms untangle more and more into an arrange- 
ment of coloured squares. 

We detect, as squares, the same colours as in the original picture. We also 
detect some that were hidden before. DeviceX is like a slider, i.e. an instru- 
ment we use to set one parameter to a certain value. The slider’s relative posi- 
tion usually indicates the parameter’s value along a scale from O to 1. 

DeviceX also functions according to this scheme, with one important 
difference, however: it does not indirectly indicate the parameter’s value. It 
rather shows it directly. The device is loaded with the contents it controls. By 
looking at the slider we look into the picture. This is its X-property: the prop- 
erty of looking into (or even through) an invisible material. 

The art historian, when applying deviceX to some data content of which 
she doesn’t really know where it exists, considers herself a “user” of the 
software. The mode of use is usually called “interactive”. The interaction is 
between her and the software, deviceX. It is quite clear to her that she is not 
shifting deviceX but the mouse in her hand. But it appears to her as if she 
was directly (and not indirectly) shifting the graphic rendition on her laptop’s 
screen. The interaction between her and the computer has become so fast 


Fig. 3. deviceX. Top: Geometry 
of an image (lower left), topol- 
ogy (right) and intermediate 
state of slider (above). Bottom: 
a series of states of an image 
in transformation from geom- 
etry (left) to topology (right) 
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Fig. 4. Left: Frieder Nake, Homage a Paul Klee. Computer drawing, 1965. Right: 
Frieder Nake, Susanne Grabowski, Matthias Krauß: Spannung. Interactive 
installation. Picture taken during the opening of my exhibition at ZKM 
Karlsruhe: to the far right, Peter Weibel in discussion 
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that she can ignore the tremendous amount of calculations going on at every 
single moment. 

The art historian probably also knows that the immediacy on the screen 
is caused by a programmer. He may now be living far away, working on some- 
thing new. But once in the past, he wrote that program that now effectly 
appears like a tool in the art historian’s hand. In some metaphorical way, it 


is a tool. 


Scene 3: Medium. Two persons having fun, 
others watching, the computer: where? 


It is now 2004. We enter the Korbakow room at Kunsthalle Bremen on our 
tour of Die präzisen Vergnügen’ (“precise delights”). We observe a couple hav- 
ing fun at a bistro table (Fig. 4 right). A screen is mounted into the tabletop. 
Two small graphics input panels can be operated by using a pressure sensi- 
tive pen. The screen in the middle displays a line drawing belonging to the 
well known “Homage a Paul Klee” program (Fig. 4 left). There is a tremendous 
difference between the old algorithmic drawing of 1965 and the interactive 
installation of 2004. 


EEG 
6 The programmer to be credited here is Matthias Krauß; see Nake/Krauß/Grabowski 
2007, pp. 137-144. 

7 The retrospective show of algorithmic works of Frieder Nake from the 1960s was 
put up under the condition that he could also present four new interactive installations. 


They were the result of collaborations with a group of students. 


The algorithmic drawing displays a complex structure of straight lines. 
Some run horizontally across the entire format. Others build bundles of par- 
allel verticals. They are much shorter, and run from one of the horizontals to 
the next or next-but-one. There are also substructures of oblique lines build- 
ing rhizome-like groups. A number of circles seem to be floating on or above 
the straight-line structure. 

Each line and each structure of the static, algorithmic drawing is kept 
in its place by some mechanic force. All the hundreds of lines stick together 
keeping a graphical balance. This observation became the starting point for 
a dynamic and, in fact, interactive adaptation: each line was interpreted as 
a force, as a spring. 

A system of springs is kept together by attachment points. These springs 
are not visible. They are a metaphor for an invisible property of each of the 
lines. The visible lines are a graphic interpretation of the spring system. 

When a visitor puts the tip of one of the two available pens onto one of the 
graphics input panels, some line or point or cell is highlighted in colour. This 
feedback tells the visitor, which one of the objects he picked. Depending on 
the kind of objects, the visitor can perform a number of possible actions. 


Fig. 5. The interactive installation Spannung. Left: Two visitors oper- 
ating (display, two panels, pens; projection visible in the background) 
Centre: Display image with visitors’ identified vertices (coloured dots). Right: 
Distorted image 
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He may move the contents of one of the horizontal bands of lines. By their 
spring property, they remain attached to the horizontals at all times (Fig. 5 
for two visitors in action). 

The visitor may also change the contents of a cell (from obliques to verti- 
cals to empty). Or he may grab one of the vertices, and pull it to a new loca- 
tion. This creates the most exciting effect because the entire structure must 
follow exactly, keeping its attachments as they were before. 

The effect of applying an outside force to test the tensions inside the 
drawing becomes even more dramatic when two visitors grab vertices simul- 
taneously and pull in different directions. This is an illustration of the basic 
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characteristic of computing: objects are always double. We will describe this 
in different ways in the next section. 

Here we should add how the field of forces influences the slow motion 
of the circles mentioned above. They have displayed a very calm and slow 
motion at all times even when the lines were static. The circles thus invited 
the visitor to think about the picture in dynamic terms. 

The circles possess another unique property. Two visitors may grab them 
simultaneously. With one circle attached to the interactive tools of the visi- 
tors, they can move the circle around or change its size by pulling in different 
directions. 

When no one is operating the installation, it displays a static minimal 
picture: a black square and a black circle, slightly overlapping, somewhat 
reminiscent of Malevich’s suprematism. As soon as one of the pens is applied, 
one of the Homage a Paul Klee drawings slowly unfolds out of the square-and- 
circle. At the same time the image is projected onto the wall. 

Pictures in a museum or gallery are silent witnesses of their artist’s work, 
of their epoch, and of systematic and historic contexts. When we walk the 
halls of the museum, we may not get anything from the rich context. When 
we read a book, listen to a guide, or engage in conversation with friends, the 
situation changes, and the paintings also. 

At first sight the interactive installation seems to be similar. But it is 
ready to tell us more about itself if we interact not only mentally, but manu- 
ally as well. Interactive use of a program — today the ubiquitous mode of 
use — belongs to the characteristic view of the computer as medium. The 
exhibition in 2004 had installations that were to be used without tools. A 
camera detected passers-by and a minimalistic static projection was set into 
motion. 

The message is: I am waiting here for you to do something; you don't 
have to be instructed, just do what you see fit, and I react. The interchange 
between you and me may tell you something, but what that could be, is 
totally up to you. My aesthetics is the unfinish. I am finished as far as I am 
a technical product. But I am obeying the law of unfinish® (a funny kind of 
activity, isn’t it?) by going on and on with no end, no goal. 

This is exactly the identity of digital media. The computer has disap- 
peared. Where is it, we may wonder? Hidden somewhere, having become a 
medium. Media are ubiquitous and unobtrusive. 
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8 The term is introduced in Lunenfeld 1999, see his introductory essay Unfinished 


business, p. 7. 


Surface & Subface or the Algorithmic Sign 


The three scenes described above stand for stages in interaction related 
to computing machinery. Its acknowledged history tells us that, without 
doubt, the computer started as an automaton. This does not only apply to 
the mathematical term automaton as used in formal theory. It applies much 
more to the historic evolution of the work of machines that had no choice, 
under given economic and political circumstances, but to give rise to auto- 
matic machinery. The concept of algorithm and the paradigm of computabil- 
ity stand for this. 

The computability feature got wrapped into a tremendously beautiful 
and successful adorning layer of pseudo-tools. They are programs which no 
one must think of as programs. You start and stop them, specify their input 
parameters, and observe their outputs in the most joyful way. Never has 
machinery made use of its own capabilities so inventively, humbly, progres- 
sively, and aesthetically as computing technology. From the first bold steps 
by Alan Kay and his group at Xerox PARC in 1971 to Apple’s first Macintosh 
and its software in 1984, only a dozen years passed. Ever since no other use 
had any chance, no other mode of existence had any relevance, but the tool 
perspective. 

The economy of the time required the transformation of the computer into 
a market commodity. The concept for this was the invention of algorithmic 
tools; the mental paradigm was interactivity. As it evolved, and as miniaturi- 
sation continued at a breath-taking speed, algorithmic tools disappeared into 
the general environment. The third phase of interactivity brought back the 
original situation. In the first scene, we had seen two men interacting with 
each other - and, in the true sense of the word, there is no interaction other 
than between humans. The two, in their deliberations, occasionally turned to 
the computer to get answers to certain well-defined tasks. 

The tool phase pushed groups of humans into the background in order 
to generate the false ideology of human-computer interaction. Human inter- 
action had to be brought back artificially by inventing CSCW - computer- 
supported cooperative work. But the tool phase was necessary and of the 
utmost importance. Without mercy, it caused everybody to use computers as 
an absolute given. Nobody can now work or live anymore without a comput- 
ing machine. 

The art world has widely accepted interactive applications of comput- 
ability. It has provided another layer of wrapping paper: the transformation 
of the automaton into a medium. A great story of only forty or fifty years: The 
algorithmic revolution (Peter Weibel)! It depends on a very simple technical cir- 
cumstance. I want to mention it at least briefly, giving it two names. Whatever 
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thing you may address, choose, pick, apply, use on or in a computer, it comes 
as a sign of a special character. I call it the algorithmic sign. 

The second name I want to attach to computer things is the pair of sur- 
Jace and subface. Let me explain this use of language before I take up the 
algorithmic sign, which is the more theoretical twist. 

When we use a computer, we use a program running on the computer. 
A program - called software - is running on a computer, when a machine — 
called hardware - is executing a code. The code is a sign for the program. 
When the program is running, it generates images on the computer display 
monitor, its main output device. Those images change in extremely rapid 
sequences. Each movement of the mouse or hit ofa button changes the cur- 
rent image. 

Screen images are visible to us. The program exists for us by its name and 
the world of images it generates. We anthropomorphise the operation of the 
program. We tell each other things like: “you should see what the new version 
of X is doing! It can now take your pictures and organise them such that it 
becomes much easier for you to ...” Thinking twice about such talk reveals a 
false conception. The program is really behaving just like any other machine: 
it is carrying out exactly what we want it to do, or at least, what our param- 
eter settings force it to do. 

A metaphorical way of talking about the program’s behaviour is, never- 
theless, justified. It is justified because the program is manipulating in its 
innermost organs what we only see as the current state of affairs. From an 
outside perspective, we may collapse this into one observation. Whatever is 
to become an utterance of the program for us to perceive, must first be stored 
in the display buffer to which the image on screen is tied in a one-to-one 
correspondence. 

The screen is the surface, the display buffer is the subface of the algorith- 
mic thing that the two of us — we ourselves and the program - are engaged 
in. The algorithmic thing comes as a visible appearance for us. At the same 
time, it comes as a computable appearance to the program. Without both 
being present and being tied to each other, nothing would work the way we 
want it to work. It does not make any sense to talk about the computer image 
without keeping in mind its visibility and computability, i.e. its computable 
visibility and its visible computability. 

The computer thing is a double insofar as it is not only visible, nor is it 
only computable. It is visible in a new meaning of the word, and it is com- 
putable in a new meaning. Our thinking needs an understanding of the old 
meanings of those two concepts. Computer images are more than visualisa- 
tions of a computation, and they are more than computations of an image. 

The world of the surface and the subface that cannot but appear together 
is apt to catch exactly this: the inherent duplicity of anything happening on 
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the semiotic machine. “Semiotic” is the correct word to characterise this. The 
algorithmic sign, which I am now going to introduce, is the theoretical con- 
cept for this semiotic perspective. 

To recall from Charles S, Peirce, a “sign, or representamen, is something 
which stands to somebody for something in some respect or capacity. It 
addresses somebody, that is, creates in the mind of that person an equivalent 
sign, or perhaps a more developed sign. That sign which it creates I call the 
interpretant of the first sign. The sign stands for something, its object.”® 

Of the many definitions of the sign by Peirce, this may be the most lucent 
one. To repeat in my own words, Peirce introduces the sign as a triadic rela- 
tion, which wraps up the old dyad of something standing for some other. The 
sign, as a relation, cannot be perceived. But it must possess the feature of 
being perceivable. This feature is called the representamen. To be perceivable 
by our senses, it must be material. In some way, the representamen carries 
the sign. It carries it insofar as it gives rise to the relation that the perceiver 
is creating upon her perception. She is creating the object and the interpre- 
tant. The two together constitute what traditionally is called the meaning of 
the sign. 

In Peirce’s great analysis, the meaning of a sign depends on a culture, 
on a context, on a community. That community makes it possible by all of 
its conventions, history, habits, etc. for us to use the sign in the interests of 
communication. This general or public component of the meaning of the sign 
is the object of the sign. 

Each subject perceiving a representamen and trying to make sense of it, 
also creates a particular or private component of the meaning of a sign: the 
interpretant. If the object is the long-lasting and generally accepted meaning 
of the sign, the interpretant is its short-term, situational and individually 
generated meaning. 

Peirce thus gives us a dual way of talking about the sign: perceiving the 
red light of the traffic sign with the general and enforced interpretation of 
“Stop!” as well as the particular and deviating interpretation of “Proceed with 
great care!”. The interpretant, by the way, is itself a sign. This introduces 
the sign as a recursive concept, as a process without end. Only the prag- 
matics of a given situation force us to interrupt the infinite sign process of 
interpretation. 

Let us now take a look at what the computer does when it receives an 
input signal. The signal corresponds to the representamen of a possible sign. 
Of course we expect the computer to function well and to do exactly what 
the input signal “tells” it to do. With the rare exception of a malfunction, 
our expectation comes true. Nevertheless, the computer performs an act that 


0220200000 EEE ZZ 


9 Peirce 1955, p. 99 


106 


formally is of an interpretive nature even if the computer is not capable of 
any interpretation. It is programmed in a definite, and precise way. The pro- 
gram “interprets” the input signal, i.e. it determines the one, and only one, 
interpretation of the line of code, or the command that it is then forced to 
execute. 

The formal act of interpreting a unit of code, in the case of the program, 
reduces to a determination. Determination is the limit case of interpretation: 
finding out the one and only meaning. Thus algorithmic signs are signs in the 
usual (Peircean) sense of the word, but with an extra interpretant. We call this 
the causal interpretant to distinguish it from the intentional interpretant. The 
latter one is what the human creates.!° 

We now close the ring. The surface of any object on the computer cor- 
responds to the intentional interpretant of the computer sign. The subface 
corresponds to the causal interpretant. I am not saying that the subface is 
the causal interpretant. For my intention here is to point at a correspondence 
between two perspectives. 

The components of digital media, of semiotic entities on a computer, or of 
things we are interested in when using a computer, can only be understood 
in the world of relations, not in the world of things. This is my message in 
this essay. What is usually called the interface between human and machine, 
appears as the coupling of surface and subface. Both are machine-bound. 
Both are faces at which one process ends, and another process starts. The 
human places rather trivial components onto the surface (like mouse posi- 
tions, or menu selections). He interprets what the program delivers in a rich 
way, influenced by his intentions, interests, situation, and context. Once the 
surface is transformed into the subface, the program starts its signal proc- 
esses, which consist of chains of determinations like any other process on a 
machine. 

The miracle of human-computer interaction is that it is impossible as 
interaction in a true sense of the word. It is happening nevertheless. This 
is possible because human acts of interpretation correspond in a rich (but 
computable) way to machine operations of determination. The miracle is that 
humans were bold and intelligent enough to establish this. The miracle is not 
that machines were so intelligent to do it. 


Wee ceeded 


10 This concept is further elaborated in a chapter of the book by Peter Bogh Andersen 


and the author (forthcoming). 


Algorithmics & Aesthetics 


A very brief remark in conclusion. Since about the mid 1990s, some 
aspects of computing science have been collected under new programmes of 
study. They are often called Digital Media, or something similar. These pro- 
grammes carry the heavy burden of striking a balance between a serious and 
appropriate study of algorithmics up to the point of script programming, and 
of aesthetics of 20th century art. They need a bit of art history as well as the 
history of computing. Their questions should be directed towards an under- 
standing of the algorithmic sign in as many ways as possible - aesthetic, 
educational, and cultural. 
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1. Models of Interactivity between flows and salti! 


“Interactivity is all there is to write about: It is the Paradox and the Horizon of 
Realization.” 


Grammatologically, the Western notational system is not offering space in 
itself to place sameness and otherness necessary to realise interaction /ality. 
Alphabetism is not prepared to challenge the dynamics of interaction directly. 
The Chinese writing system in its scriptural structuration is able to place 
complex differences into itself, necessary for the development and design of 
formal systems and programming languages of interaction. The challenge of 
interactionality to Western thinking, modeling and design interactivity has to 
be confronted with the decline of the scientific power of alphanumeric nota- 
tional systems as media of living in a complex world.? 

The challenge I see for media artists is not only to develop interactional 
media constellations but also to intervene between the structures and dynam- 
ics of interactional systems as international corporations, governments, mili- 
tary and academia force them on us.? 


1.1 Comparison of two approaches to interactivity 


This paper takes the risk to compare two fundamentally different 
approaches to interaction and reflection in computational systems: Milner’s 
bigraphs and diamond theory. Milner’s bigraph model and theory of interac- 
tion is highly developed, while the diamond model applied to this interac- 
tional scenario and confronted with the bigraphs model is presented here for 
the first time. 

The Milner model is presupposing a world-view (ontology, epistemology) 
of homogeneity and openness. Its basic operation is composition in the sense 
of category theory. Composition is associative and open for infinite iterability. 
Milner’s model is a model of interaction in a global sense but it is not thema- 
tising formally the chiastic interplay of local and global aspects of interaction. 
Its merit is to have developed a strict separation of topography (locality) and 
connectivity for a unifying theory of global and mobile interaction (ubiquitous 
computing) surpassing, in principle, the limits of Turing computability. 


ALLL 
1 Thanks to Marianne Dickson, Edinburgh, for bridging the corrections and correct- 
ing the bridges of this composition. 
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In contrast, the diamond model, which is just emerging,* is based on an 
antidromic and parallactic structure of combination of events in an open/ 
closed world of a multitude of discontextural universes. In such a pluri-versal 
world model, each composition is having its complementary combination. 
With that, iterability for diamonds is not an abstract iterativity but interwo- 
ven in the concrete situations to be thematised, and determined by iterative 
and accretive repetitions, involving their complementary counterparts, with- 
out a privileged conceptual initial/final object. 

This leads to a theory of diamonds as a complementary interplay of cate- 
gories and saltatories (jumpoids) with the main rules, globally, of complemen- 
tarity and locally, of bridging. Diamonds are involving bi-objects belonging 
at once to categories and to saltatories, ruled by composition and saltisition 


(jump-operation).® 


1.2 Interactionality as interplays 
between categories and saltatories 


In less technical terms, the polycontextural approach of diamond theory 
is supporting three new features: 

First, it supports the idea of irreducible multi-medial contextures and their 
qualitative incomparability. That is, different media like sound, video, picture, 
text, graphics, etc., are conceived as logically different and as organised and 
distributed conceptually in a heterarchical sense. To thematise media as a 
digital contexture is not more than to emphasise their informatical and physi- 
cal aspect, which is as such a contexture, too. 

Second, it supports the possibility of mapping the (outer) environment of 
a contexture (media) in itself, i.e., to offer an inner environment for reflection- 
ality. Contextures, to be different from systems, have to reflect their environ- 
ment into their own domain. Hence, a contexture has to be understood as 
being involved into interplays of inner and outer environments. 

Third, it supports the possibility of simultaneously realising movements 
(actions) and complementary counter-movements on a basic level of concep- 
tualisation and formalisation. If composition of events inside a contexture, 
and mediation of different contextures to a compound contexture, polycon- 
texturality, are characterised by the rules of combination, i.e., identity, com- 
mutativity and associativity, a new feature of composition is discovered by the 
diamond approach, which is antidromic and parallax, corresponding struc- 
turally to the otherness of the categorical system. 
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Therefore, the questions of interactionality in a diamond framework are 
not primarily, how do we globally move, physically and informatically, from 
one topographic place to another, but how do we move by interaction from 
one medium to another medium of a complex knowledge space. With the 
appearance of the semantic web and knowledge grid® such developments are 
unavoidable. Obviously, the polycontextural diamond approach is not opt- 
ing for a principally homogeneous global field of informatical and physical 
events but for a discontexturality of different media, situations, contexts of 
meaning. 

The Milner Model is well based, principally, on category theory, the dia- 
mond model has to develop its own new formalism, risked here as a dia- 
mondisation of category theory. Hence, both theories are in a constellation, 
which offers a reasonable possibility for comparisons. 

Because the bigraph model is based on category theory and its concept 
of composition with its abstract iterability, the diamond model also has to 
develop a distinct concept of composition (combination), one which involves 
a complementarity of at least two different concepts of composition, i.e. the 
categorical and the saltatorical, and which is opening up the operativity ofan 
open/closed concept of iter /alterability. 

Even if only metaphorically and still vague, what is common to both mod- 
els is their dichotomous, dual, complementary and orthogonal approach to 
interaction and interactionality. The Milner model is focused on message 
passing, flow of informatic objects, the diamond model on agents and their 
reflectional/interactional activities with an emphasis on intervention. 


2. Milner’s bigraph model of interaction 


Out of his cloud of keywords to ubiquitous computing and interactivity, 
Milner chooses at his Beijing 2005 performance 3 leading features: locality, 
mobility and connectivity.’ 


2.1 Locality and connectivity 


Locality 


“Programming the digital computer ramifies the use of space and spatial 
metaphor, both for writing programs and for explaining why they work. This 
shows up in our vocabulary: flow chart, location, send and fetch, pointer, 
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nesting, tree, etc. Concurrent computing expands the vocabulary further: dis- 
tributed system, remote procedure call, network, routing, etc. 

We are living with a striking phenomenon: the metaphorical space of 
algorithms — graph, array, and so on - is mixed with the space of physical 
reality.”® 


Physical and virtual space 


“Informatic objects flow in physical space; physical objects such as mobile 
telephones manipulate their informatic space.” 


“The picture illustrates how physical and virtual space are mixed. It rep- 
resents how a message M might move one step closer to its destination. The 
three largest nodes may represent countries, or buildings, or software agents. 
In each case the sender S of the message is in one, and the receiver R in 
another. The message is en route; the link from M back to S indicates that the 
messages carries the sender’s address. M handles a key K that unlocks a lock 
L, reaching an agent A that will forward the message to R; this unlocking is 
represented by a reaction rule that will reconfigure the pattern in the dashed 
box as shown, whenever and wherever this patterns arises.”° 

“Bigraphical reactive systems are a model of information flow in which 
both locality and connectivity are prominent. In the graphical presentation 
these are seen directly; in the mathematical presentation they are the sub- 
ject of a theory that uses a modest amount of algebra and category theory. 
A bigraph may reconfigure both its locality and its connectivity. The example 
pictured above shows how reconfiguration is defined by reaction rules; in that 
case, the rule may be pictured thus: 

The mathematical structure of bigraphs allows concepts to be treated 
somewhat independently; for example, connectivity and locality are treated 
orthogonally.”!° 
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“So the challenge to bigraphs is to provide a uniform behavioural theory, 
allowing many process calculi to be expressed in the same frame while pre- 
serving their treatment of behaviour.”!! 


The aim of a new design 


“The challenge for global ubiquitous computing is to devise theories and 
design principles in close collaboration, ...”'? 

“The long-term aim of this work is to provide a model of computation on a 
global scale, as represented by the Internet and the World Wide Web. The aim 
is not just to build a mathematical model in which we can analyse systems 
that already exist. Beyond that, we seek a theory to guide the specification, 
design and programming of these systems, to guide future adaptations of 
them, and not to deteriorate when these adaptations are implemented. [...] 

This will only be achieved if we can reverse the typical order of events, in 
which design and implementation come first, modelling later (or never). For 
example, a programming language is rarely based thoroughly upon a theo- 
retical model. This has inevitably meant that our initial understanding of 
designed systems is brittle, and deteriorates seriously as they are adapted. 

We believe that the only acceptable solution, in the long run, is for sys- 
tem designs to be expressed with the concepts and notations of a theory rich 
enough to admit all that the designers wish.”'? 


2.2 Strategies of orthogonal simultaneity 


“So our strategy here is to tackle just two aspects — mobile connectiv- 
ity and mobile locality — simultaneously. In fact this combination contains a 
novel challenge: to what extent in a model should connectivity and locality 
be interdependent? In plain words, does where you are affect whom you can 
talk to? To a user of the Internet there is total independence, and we want 
to model the Internet at a high level, in the way its connectivity appears to 
users. But to the engineer these remote communications are not atomic, but 
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represented by chains of interactions between neighbours, and we should 
also provide a low-level model, which rejects this reality. So we want to have 
it both ways; furthermore, we want to be able to describe rigorously how the 
high-level model is realised by the low-level one.”!* 


Milner’s Model of bigraphs’® 
RESOLVING A BIGRAPH INTO PARTS 


O/a 


place graph link graph 


bigraph 


sites t ry inner names 


2.2.1 Statics of interaction: Categorical framework 


“Abstract. This paper axiomatises the structure of bigraphs, and proves 
that the resulting theory is complete. Bigraphs are graphs with double struc- 
ture, representing locality and connectivity. They have been shown to repre- 
sent dynamic theories for the pi-calculus, mobile ambients and Petri nets, in 
a way that is faithful to each of those models of discrete behaviour. While the 
main purpose of bigraphs is to understand mobile systems, a prerequisite 
for this understanding is a well-behaved theory of the structure of states in 
such systems. The algebra of bigraph structure is surprisingly simple, as the 
paper demonstrates; this is because bigraphs treat locality and connectivity 
orthogonally.”!® 


2.2.2 Dynamics of interaction: 
Labeled process calculi 


“Let us repeat: in a pure bigraph G : <m, X> -> <n, Y> we admit no asso- 
ciation between its outer names Y and the roots (regions) n, nor between the 
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inner names X and the sites m. It is this dissociation that enables us to treat 
locality and connectivity independently, yielding a tractable theory.”'” 

The dynamics of bigraphs is formalised by labeled process calculi: 

“The challenge from process calculi is to provide a uniform behavioural 
theory, so that many process calculi can be expressed in the same frame 
without seriously affecting their treatment of behaviour. We now outline how 
research leading up to the bigraphical model has addressed this challenge. 

It is common to present the dynamics of processes by means of reactions 
(also known as rewriting rules) of the form r —> r’, meaning that r can change 
its state to r’ in suitable contexts. In process calculi this treatment is typically 
refined into labelled transitions of the form a ->! a’, where the label 1 is drawn 
from some vocabulary expressing the possible interactions between an agent 
a and its environment. These transitions have the great advantage that they 
support the definition of behavioural preorders and equivalences, such as 
traces, failures and bisimilarity. But the definition of those transitions tends 
to be tailored for each calculus.”!® 


2.2.3 Formalisation of interaction: 
Bigraphs as tensor categories 


“This chapter establishes place graphs, link graphs and bigraphs as 
arrows in certain kinds of category. Any kind of category is concerned with 
operations upon arrows, especially composition.”!° 

“Note that this combination is quite distinct from the categorical composi- 
tion used to insert one bigraph into another (e.g. an agent into a context). But 
it is simply related to them; to compose two bigraphs categorically, we first 
resolve them into their respective place graphs and link graphs, then compose 
these, and finally combine the results into a new bigraph.””° 


2.2.4 Axiomatics of bigraphs 


“The topic of this paper is to axiomatise the resulting structure of bigraphs. 
The justication for such a specific topic is threefold. 

First, the work already cited gives ample evidence that a graphical struc- 
ture combining topography with connectivity has wide application in com- 
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puter science; for as we have seen it brings unity to at least three models of 
discrete dynamics, each of which has already many applications. 

Second, it appears that the algebraic treatment of such dual structures 
has not been previously addressed; yet the behaviour of systems whose con- 
nectivity and topography are both reconfigurable may be so complex that 
their dynamics cannot be properly understood without a complete and rigor- 
ous treatment of their statics. Bigraphs are just one possible treatment of 
such dual structure, but it is likely that their static theory can be modified 
for other treatments. 

Third, as we shall see, dual structures seem to require a novel kind of 
normal form which is essential to a proof of axiomatic completeness.”?! 


Axiomatics (Table 1) 


CATEGORICAL AXIOMS: 
lid = A = idA 
UBC) (AB)C 


A® id, 4 id A 
1® (BC) (A®B)@c 
(A By)( A By) (A; Ao) ® (Bı Bo) 
Ya = id; 
VIVES id; 
y.K(A® B) (B® A)yu.s (A: H-1,B: J> K) 
LINK AXIOMS: 
you 1 
yoy id, 
Yoy idy u/X) =/(YwWN) 
PLACE AXIOMS: 
merge(1 © idı) idı (unit) 
merge( merge idı) merge(id, merge) (associative) 
mergeyı.ı merge (commutative) 
NODE AXIOMS 
(idi ®a)Kz = Kaiz- 


“In other words, the axioms are both sound and complete. They say sim- 
ple things: The place axioms say that join is commutative, has a unit and 
is associative; the link axioms say that the formation of links obeys obvious 
rules; the node axiom says that we can name ports arbitrarily. ”?? 


2.2.5 Completeness of the axiom system 


“The completeness of the axiom system in Table 1 depends primarily on 
two things: first, that all linking can be exposed at the outermost level of an 
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expression; second, that we have a strict symmetric monoidal category of 
bigraphs, with a tensor that is partial on objects. Crucial to the tensor is that 
it is bifunctorial, i.e. (A, x B A, x B,) = (A,A,) x (BB); this axiom underlies 
most of our manipulations. Thus the discrete normal form, DNF, has been 
crucial for the proof of completeness.”?® 


2.3 Orthogonality of topography and connectivity 


2.3.1 Underlying world model 


The bigraph model of interaction is highly flexible and is liberating fur- 
ther research from unnecessary fixations. Bigraphical reactive (re-writing) 
systems as models of information flow are dealing with locality and connectiv- 
ity as orthogonal events, distributed over two dimensions. Such a separation 
of structural locality and behavioural connectivity enables a clear modeling 
and an effective formalisation as a bigraph or bipartide system. Spaciality is 
conceived as static, formalised by category theory and behaviour as dynamic, 
formalised by process calculi (pi-calculus). 

The bigraph model of interaction seems to belong to a world model with 
the characteristics of: “Everything in this world is changing but the world in 
which everything is changing doesn’t change.”** Ubiquitous and global com- 
puting is presupposing an epistemologically uniform, homogeneous and 
unique world of physical and informatical events. 

Diamond theory can be set in some kind of a correspondence with a bipar- 
tide model but it is turning to a world model where there are many worlds in 
which things are changing and in which worlds themselves are changing too. 
Diamond Theory is involved not in a new super-stable world but in the game 
of interactionality/reflectionality between worlds and events, hence enabling 
system designers and media artists to intervene in and between those worlds 
guided by the metamorphic dynamics of polycontextural diamonds. 

Messages in the diamond model are conceived as polycontextural and 
as belonging simultaneously to different contextures of irreducible kinds of 
meaning. Message passing in such a model is not done by the metaphor of 
key/lock/unlock/agent in a location/connectivity setting because a key in 
this pluriversal world-model appears always as necessarily polysemic and its 
acceptance has to be negotiated by reflectional and interactional activities. If 
such complex transactions are becoming stable in their usage, a reduction to 
the mono-contextural key-model can be introduced by reducing complexity. 
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2.3.2 Chiastic transition metaphor 


Hence, in a chiastic metaphor, we can state that statics in the bigraph 
model becomes dynamics in the diamond model; and dynamics becomes 
statics in the diamond setting because its dynamics is bracketed and moved 
into a multitude of process-structures wherein the dynamics of the differ- 
ent behavioural systems have an arena in which to act. Therefore, category 
theory as formalism for interaction has to be dynamised towards diamond 
theory. That is, category theory has to be diamondised towards a dynamic 
structural formalism, which is an operational structuration. 


2.3.3 Opting for an interventional design 


The British Grand Challenge project for computing is not touching the 
principle hierarchy between mathematics and informatics. Since the Greeks 
time has changed and a reversion and displacement of this hierarchy might 
be the grand challenge of a new understanding of global computing.” 

From a model of interactions to a design of interactionality, the transitions 
to be risked might be: 

From the global, ubiquitous and universal web of computation, to the 
kenomic grid of pluriversal contexturality, containing the chiasm of global/ 
local scenarios. 

From the locality in the Actor model of informatical events to the position- 
ality of contextures in the kenomic grid, positioning informatic localities. 

From the mobility in the Actor model of informatical flows between ambi- 
ents (context, locality) of the same contextural (ontological, logical, semiotic) 
structure to a metamorphosis between contextures, augmenting complexity/ 
complication of contextural scenarios implementing clusters of informatical 
ambients and mobility. 

From the operations between actional ambients to the operationality in 
polycontextural situations realised by the super-operators (identity, replica- 
tion, permutation, reduction, bifurcation) placing ambient operations into the 
grid. 

From the connectivity of actions at a locality of message passing, using 
a key to unlock a lock of an agent, to different kinds of mediation between 
contextures containing informatical connectivity. 
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These transitions seem to record a catalogue of minimal conditions to be 
fulfilled to realise interactionality/reflectionality and interventionality in such 
complex constellations as the emerging knowledge grid.” 


3. Diamond theory of interactionality 
3.1 Diamond Strategy 


Encounter 


Diamond strategies are sketching transitions from the mail model of 
interaction in bigraphs to the encounter model of interactionality/reflection- 
ality and intervention. 

Before we can play the bipartide game of locking and unlocking (by pass- 
ing a key in a structure of orthogonal locality and connectivity to reach an 
agent capable of passing the message to another agent), the otherness of the 
actors involved has to be acknowledged and accepted by all the interactional 
activities of the actors involved. 

It can be described as the action of addressing an addressee, which is 
able to accept the addressing by offering its own addressable structure. After 
having been addressed and having the addressing accepted by the addressed 
and after the addresser has recognised the acceptance of being addressed and 
the addressing is thus established, information can be exchanged between 
agents in the sense of communication.” 

Interactivity in the encounter-model, therefore, is conceived as a mutual 
action of acceptance and rejection between different agents. Only on the basis 
of this interactional agreement can information exchange happen.”® 

Therefore, the structure of interaction is always complex: at once realising 
the addresser and the inner environment of the addressee. This simultane- 
ity of inner and outer environments of agents involves a kind of structural 
bifurcation and mutual actions of acceptance and/or rejection of the involved 
agents based on the complexity of their architectonics. That is, the addressee 
has to give space (einräumen) to the addresser to be addressed. To address 
and to accept to be addressed is a mutual action of at least two agents in a 
common co-created environment. Hence, the actional structure of interac- 
tionality is not only bipartide but antidromic, too. This phenomenon forces a 
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formalisation paradigm beyond mathematical category theory, which finds a 
very first attempt to a realisation in the proposed diamond theory.”° 


Intervention 


An interaction of an agent, including reflections on the behaviour of a 
partner agent, which is intended to change the meta-rules of the partner- 
agent can be called an intervention. An agent is intervening into an interac- 
tion in attempting to change the meta-rules of the agent. An intervention 
takes place if an agent is interacting with another agent in a way that the 
agent is forced to change his meta-rules to stay in the game of computation 
and interaction.?® 


The aim is not just to build a mathematical model in which we can analyse 
systems that already exist. Beyond that, we seek a theory to guide the 
specification, design and programming of these systems, to guide future 
adaptations of them, and not to deteriorate when these adaptations are 
implemented. There is much talk of the vanishing ubiquitous computer 
of the future, which will obtrude less and less visibly in our lives, but 
will pervade them more and more. Technology will enable us to create 
this. To speak crudely, we must make sure that we understand it before it 
vanishes.?! 


Diamond strategies are not only asking for an understanding of such 
trends, like the vanishing of computational challenges for users by ubiqui- 
tous computing, but for the possibility of intervention by computer designers, 
scientists and users into such trends. Thus, opening up interplays between 
users and general computation, avoiding any kind of regression into eupho- 
ria, criticism and luddism of humanistic self-defence. 


3.2 Towards Diamond Theory 


3.2.1 From categorical 
composition of morphisms to diamonds 


Actions from A to B can be considered as morphisms, symbolised by an 
arrow from A to B, A -> B. In this sense, morphisms are universal, they 
occur everywhere. But morphisms don’t occur in isolation, they are composed 
together in interesting complexions. The composition of morphisms (arrows) 
is defined by the coincidence of codomain (cod) and domain (dom) of the mor- 
phisms to be composed, called the matching conditions (MC). That is, (f, g) 
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is composed (f o g) iff cod(f) = dom(g). This highly general notion of morphism 
and composition of morphisms is studied in Category Theory.*” 

A general descriptive explication of the concept of composition of mor- 
phisms is given by the following diagram. It contains the table of the match- 
ing conditions. Here, the distinction between objects, A, B as domain and 
codomain properties of morphisms, and the alpha (a) and omega (w) function- 
ality of morphisms are included. 


Matching Conditions 


Descriptive Composition w, Ya, 
(A, a) —— (e, o) 0(4?, a) le o) fate 
N a (A, a,)- (A, a,) 
(A, a) —=— (5, o,) (B?, @, ) Ey (8°, w) 


Hence, not only the codomain B! and the domain A? as objects have to 
coincide, but also the actional domain “alpha,” (a) and the actional codo- 
main “omega,” (Œœ) as functional properties of the morphisms f and g, have to 
match. Obviously, the commutativity of the diagram has to fulfill, addition- 
ally, the matching conditions for (A!, a) with (AS, a) and (B?, o,) with (BS, %,), 
defining the composition (f o g). 

First, without the actional alpha/omega-notation we get the matching 
conditions, coincidences, for categorical composition based on the objec- 
tional distinction of domains and codomains. 

Second, stripped off of the set-theoretical or objectional content of the 
domains and codomains of morphism, the functionality of beginnings (a) and 
endings (œw) remain. Composition then means an exchange relation between 
the ending of a morphism and the beginning of another morphism, i.e., 
between (w J and (a). Both founded in the coincidence relation between the 
actional domain of the first and the actional codomain of the second mor- 
phism, establishing the commutativity of “object-free” categorical composi- 
tion, i.e., the morphism between (a) and (w,), i.e., (,) > (w,). 


id N Z diff 

a, u, ta, iso, 

| coinc comp | 
morph h 


a > wW. 
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Such a chiastic approach, emphasising the pure functionality of composi- 
tion uncovers the possibility of a new relationship involved in the definition 
of actional composition: the complementarity of the commutative morphism 
between the beginning (a2) and the ending (w1) involved in the categori- 
cal composition, building the “antidromic and parallax” hetero-morphism 
between (a4) and (w4), i.e., (a4) —> (w4). 

Hence, functional composition of morphisms, which are represented by 
order relations, is based on the functional matching conditions, MC, of two 
types of relations: exchange and coincidence relation building together with 
the order relations, a chiastic pattern in form of a diamond. Obviously, this 
singular diamond is occupying a place and is localised in a grid of diamonds 
and thus ready to be disseminated. 

Third, both thematisations together, the objectional and the actional, 
with morphisms and hetero-morphisms, define the diamond composition of 


morphisms. 


3.2.2 Diamond model of system/environment 


Some wordings to the diamond system/environment relationship might 
be listed: 

What’s my environment is your system. 

What’s your environment is my system. 

What’s both at once, my-system and your-system, is our-system. 

What’s both at once, my-environment and your-environment, is our- 
environment. 

What are our-environments and our-systems is the environment of others- 
system. 

What’s our-system is the environment of others-system. 

What’s neither my-system nor your-system is others-system. 

What’s neither my-environment nor your-environment is others- 


environment. 
Diamond-Obj ®: Obj © -> Obj ® 
system my envm 


system (3 aı prop > I w4 envim 


our Ic | PS | ef others 


envm w3 (O > Oh? a4 system 
opp 


envm your system 
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Some environement / system formulas 
+ :and, |: neither - nor, =: not. 


my(syst, env) = your (env, syst) 

our (syst, env) = others (env, syst) 

my (syst, env) + your (syst, env) - our (syst, env) 
my (syst, env) | your (syst, env) - others(syst, env) 
our (env + syst) = others(env) 

others(env + syst) = our (syst) 

our (env | syst) = others (syst) 

others|env | syst) - our (syst) 

my (syst) = >(your (syst), others ( syst)) 

your (syst) = „(my (syst), others(syst)) 

our (syst) = ~(my (syst), your (syst), others(syst)} 
others(syst) = „(my (syst), your (syst), our (syst)) 


The diamond modeling of the otherness of the others incorporates the 
otherness into its own system. An external modeling of the others would have 
to put them into a different additional contexture. With that, the otherness 
would be secondary to the system/environment complexion under considera- 
tion. The diamond modeling is accepting the otherness of others as a “first- 
class object”, and as belonging genuinely to the complexion as such. 

In another setting, without the “anthropomorphic” metaphors, we are dis- 
tinguishing between a system and its internal and its external environment. 
The external environment corresponds to the rejectional part, the internal to 
the acceptional part of the diamond. Applied to the diamond scheme of dia- 
mondised morphisms we are directly getting the diamond system scheme out 
of the diamond-object model. 

Much work has been done on interactionality/reflectionality and inter- 
ventionality/interlocutionality on the basis of polycontextural notions and 
formalisms.** Despite its chiastic and proemial approach, this work did not 
yet include the others-system of the diamond model. 
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3.3 Diamond Structuration 


Diamonds in this sketch are conceived as interplays between categories 
and saltatories based on morphisms and hetero-morphisms with their com- 
positions, saltisitions and bridgings. Saltatories are the complementary con- 


cept of categories. 


Conceptuality 
saltisition 


ER 


objects arrows 


N... 


composition 


The conceptuality of diamond theory is introduced by an application 
of the diamond strategies to the basic concepts of category theory: objects 
and morphisms (arrows). Objects are understood in this setting as proposi- 
tions, arrows as oppositions. Compositions appear as the both-at-once of 
objects and arrows, and saltisitions as the neither-nor of objects and arrows. 
Composition and saltisitions, hence, are complementary concepts. 

saltisition, saltatory 

salto mortale: jump from the apriori to the empirical (Immanuel Kant). 

diamond strategies: double salto mortale from the theoretical to the hyper- 


theoretical. 
Diamond Theory Diamond duality 
Category : A =(Obj*, hom, id, o) Category | Saltatory 
Saltatory : a = (Obja , het, id, | ) 


pth -([a; a], compl, diff, .) Cat | Cat” | Salt | Sa 


Categories are dealing with composition of morphisms and their laws. 
Saltatories are dealing with the jump-operation (saltisitions) of hetero-mor- 
phisms and their laws. Diamonds are dealing with the interplay of catego- 
ries and saltatories. Their operation is interaction realised by the bridging 


operations. 
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The laws of identity and associativity are ruling compositions, as well 
as saltisitions. Complementarity between categories and saltatories, i.e., 
between acceptional and rejectional domains of diamonds, are ruled by dif- 
Jerence operations. Duality operations are applicable to both, categories and 
saltatories. 


Commutativity and associativity 


Commutativity Condition 
Iff,geMC, IE MC: 
then 


9 )f=(gof)il/ 


wf) ß alg) 
with diff (a (9)) - a(i) 
arr(o(f))-o() 
such that 
A——B 
of Y t9 Ib, -— b, 
c 


bi - commutes. 


Associativity Condition 

Iff,g,h,k EMC andl,m, ne MC: 

then 

(k Ohogd f) - 

ko (ho (g of)) =ko ((r og)o f) 
((kohlog)of-(kol(hog)) of [1 (mn) - (Hm) n 
(kohlo(gof)-ko(hog)orf 


3.3.1 Identity and difference 


“This shift becomes even more apparent if one examines the foundational 
concepts Nishida develops later in his career, the ‘self-identity of the absolute 
contradiction’ and the ‘many in one, one in many’ (tasokuitsu, issokuta); the 
former can be paraphrased as the ‘identity of absolute difference’ and the lat- 
ter as ‘plurality in oneness, oneness in plurality’.”** 
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Identity and difference morphisms 


bi- Object [x, x] 
id 

x E Salt 

T diff e Diam 
X € Cat 

id 


Identity is a mapping onto-itself as itself. 

For each object X of a category an identity morphism, ID, xp which has 
domain X in the category and codomain X in the same category exists. Called 
ID, or id, for ID, xr 

For each object x of a saltatory an identity morphism, ID, ,,, which has 
domain x in the saltatory and codomain x in the same saltatory exists. Called 
ID, or id, for ID, xr 

Difference is a mapping onto-itself as other. 

For each object X of a category a difference-morphism DIFF x „ which has 
domain X in the category and codomain x in the saltatory exists. 

For each object x of a saltatory a difference morphism, DIFF «xp which has 
domain x in the saltatory and codomain X in the category exists. 

This wording is a strict paraphrase of the common wordings of category 
theory. It also emulates its architectonics: from objects to morphisms to 
isomorphisms and to natural transformation, etc. Nevertheless it is not yet 
reflecting the reversed architectonics of the diamond way of thinking, where 


objects occur last and not first. 


Identity and difference composition 


ID and DIFF composition 


Identity Difference 
vf, X, Y,0 €Cat: Om Cat, Salt € Diam : 
fo, ID, =f=ID, 0, f. |v[X, x], [v, y] © Diam 
VI, x, y, || © Salt: [f, 1] (0, I) 2 DIFF, 
IL ID =-1=ID ||. 
Zr a ey -[f, 1|- 
DIFF, p (eo) ier} 
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3.3.2 Diamond concepts 
between iso- and xenomorphism 


“One philosophical reason for categorification is that it refines our concept of 
‘sameness’ by allowing us to distinguish between isomorphism and equality.”?° 


Isomorphisms 

Isomorphism in Cat: Cat, 

vf, g €Cat: 

F [gof = id 

Xz ty iff |9° Ox 
d [f o g = id,. 

Isomorphism in Salt: Salt,, 

vi, me Salt: 


j |! || m= id,. 
Diamond Isomorphism: Diam, 


Om Cat, Salt € Diam: 
right - domain -ISO : 


x (go f)* id, = id, 
Taiff iff |` U 
x—tey id, «(f o g) = id, , 


left- codomain- ISO: 


y (gof) eid - id, , 

Taiff | iff |* 4 
xey id, e (f og) = id, , 
Hetero- ISO 


right - domain -ISO : 
x——y 
tT diff iff | 
` lid, «(m |l I} = id, 
(xX . ‘as 
left - codomain - ISO: 

\ 
hr ui (miii) e id, -id, , 
Taiff | iff , 
Y 


id, ¢(1\| m) = id, , 


Category theory is studying, at first, isomorphisms between objects as 
domains and codomains of morphisms, then the trip goes on with functors, 
natural transformations and so on. Their basic element, thus, is an elemen- 
tary, single morphism and their basic operation is a single identity morphism. 
Diamond theory is dealing with the interplay between categories and saltato- 
ries, hence, the elementary situation is not a single morphism but the inter- 
action of the selected morphism and its two corresponding, i.e., interacting 
hetero-morphisms based on identity and difference operations. That is, the 
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domain and the codomain of the selected morphism has to consider the cor- 
responding domain and codomain of the hetero-morphisms involved. This is 
ruled by the difference operation. 

Hence, the isolated objects as domains and codomains have to be sup- 
plemented by their own counter-parts, codomain and domain, to build their 
hetero-morphisms. In other words, the full interplay of morphisms, identity 
and difference mappings, has to be involved to realise proper diamond iso- 
and xenomorphisms. 

Full combined isomorphisms between morphisms and hetero-morphisms 
are naturally constructed out of the partial iso- and xenomorphisms.’® 


3.3.3 Diamond concept of transversality 


A difference-philosophical interpretation of transversal isomorphisms 
could be found in the classical formulations of “The identity of oppositions, 
i.e., the identity of difference and identity.” and “The difference of identity and 
difference”. Both formulations are in some sense dual. 

Further, more complex isomorphisms are easily composed by a combina- 
tion of right- and left-isomorphisms. 


Transversality ISO 
trsv, : diff (A) (B) 
trsv, : A > diff (B). 


right - transversal -ISO 


ier. trsv,  odiff, „= id, 
Taiff Strsv | iff se 
ae trsv, p e (f 0g)= dif, 


left - transversal- ISO : 


y (g of) e trsy, „=diff, | 
Arsv | diff | iff 4 ” 
y trsv,, , © diff, j = id, 


X 


3.3.4 Facets of diamond isomorphisms 


The concept of diamond isomorphisms is not solely dynamising the realm 
of sameness, as is the aim of category theory, but it is also inter-wined with 
the differentness and strangeness of otherness.?” 
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1. Sameness (up to isomorphism) 
Vf, 9, X,Y €Cat: 
bap fs gof=id,, 
X Y iff j 
9 fog= id, 5 


2. Differentness (up to xenomorphism) 
vf, X, Y E Cat, YI, x, y E Salt : 


x——y | [lef = diff, . 
diff T T diff | iff 
pee es fel=diff , 


3. Strangeness (up to heteromorphism) 
Vf,g, X, Y E Cat, Vl,m, x, y E Salt: 


x=") lo oF) +(mui)=id,., 
T diff iff |, \ 
(f o g) « (11l m) = id, , 


X= iy 


3.4 Interactionality as interplays in diamonds 


Interactionality of diamonds studies the interaction between dissemi- 
nated categories and saltatories of polycontextural diamond systems. Given 
contextures in isolation, topics like duality and complementarity in diamonds 
are interactional, but they do not yet considering the inter-twining and inter- 
vening properties of interactivity as it happens with bridging. Thus, interac- 
tionality as an intra-contextural interplay occurs in elementary diamonds in 
forms of duality, complementarity, bridging and distributivity. 


Duality for Categories: “two for the price of one” 


The Duality Principle for Categories states 
Whenever a property P holds for all categories, 
then the property PP holds for all categories. 
The proof of this (extremely useful) principle follows immediately from the 
facts that for all categories A and properties P 
(1) (Ar)P = A, and 


(2) P(A) holds if and only if P(A°) holds.** 


Duality is defined for diamonds as duality of categories and duality of 
saltatories. 
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Complementarity of formal languages 


The general principle underlying these limitations was called the linguistic 
complementarity by Loefgren. It states that in no language (i.e. a system for 
generating expressions with a specific meaning) can the process of inter- 
pretation of the expressions be completely described within the language 
itself. In other words, the procedure for determining the meaning of expres- 
sions must involve entities from outside the language, i.e. from what we 
have called the context. The reason is simply that the terms of a language 
are finite and changeless, whereas their possible interpretations are infinite 
and changing.*° 


The double meaning of diamond objects, bi-objects, is complementary 
and in their orientations they are not parallel but antidromic and deferred 


regarding the complementary system. 


Bridging categories and saltatories 


f h 
a, —o J w, Ja, —— a, 
fg > 
D. 
a, c 3 
għ > (i 
a, Os 
foh 
> 
a w, 


Bridging is not an operation of mediation or switching of and between 
diamonds or acceptional and rejectional actions in diamonds, but an opera- 
tion to knot the two realms together, the categorical and the saltatorical. In 
the diagram, between the hetero-morphism k, l, the morphism g is offering a 
bridge, marked in red, and thus interacting between the saltatorical and the 
categorical domain of the diamond. Complementarily, the two bridge pillars 
of the bridge are offered by the two hetero-morphisms l, k defining the bridge- 
work g. Thus, bridge and bridging are complementary actions, too. Both are 
reflecting the complementarity between categories and saltatories. 


Distributivity of composition, saltisition and bridging 
Because diamonds are based on interplays between categories and sal- 


tatories, which are involved with two fundamental operations: composition 
(o) and saltisition ( | | ) with bridging (°) too, it is reasonable to find interactive 
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laws as distributivity between those basic operators inside the very definition 


of the conception of diamonds. 


3.4.1 Duality in diamonds as 
duality in categories and saltatories 


Duality in Diamonds 
duality in categories 


| duality in saltatories 


(gof)-A>c 

dual(g o f) = dual (dual(B—C) o dual(A — B)) 
= dual ((B — C) o (A= B)) 
- ((4=8) 0(8~c)) 
=(A-B=C) 
-A=C. 

Hence, ((9 of\=A -c)e Cat 

iff 

(dual(g o f) = A- C) E Cat”. 

X =g 0f -=|{g of); u]: 

X € Cat iff dual(x) € Cat” 


u = (w, — a, ) = compl (g o f) 
dual (compl(g o f)) ~ dual (u) 
dual(u) = dual(w, — «,) 
wild ah, 
compl (dual (g o r)) = compl |f o g) =(«, =o). 
Hence, (u = (w, = «,)) € Salt 
iff 
(dual (u) =a, > o,) € Salt”. 


|X € Salt iff dual(x) E Salt” 


3.4.2 Complementarity of categories and saltatories 


Complementarity of Acc and Rej 
X € Acc iff compl (x ) E Rej 


X=gof: 


1. X € Acc if compl (X) E€ Rej 


compl (g o f) = compl (compi(9) o compi (f )) 
= compl (diff (cod (f )) o diff (dom(g)}) 


= compl ( B.) o (B..))- oa, 


(u 50, a, ) E Rej 


Hence, (g o f) € Acc if (u: w, — a,) € Rej 


(9 o f) € Acc if (sor) E Rej. 


2. comp! (x) E Rej if X € Acc 


compl (o, - a,) = compl (comp! Q ) -— wae )) 


= compl IM 


= (Amm 


a 


Cora ) 


B om 


> Cog ) 


Bom 


= (f o Aia 
3. Hence, X E Acc iff compl (X) E Rej. 
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3.4.3 Bridging between categories and saltatories 


This new feature of bridge/bridging is ruling concrete intrinsic 
interactions. 


Bridging conditions and associativity for interactions 


Bridge and Bridging Conditions BC 
1. Wk, |, n€ HET, Vf,g,he MORPH : 
a. composition 
gof,goh, 
(hog)of,ho(gor)emc, 
b. saltisition 
Ik, nil, u 
nil (! ik), (n II 1) I KEMC, 
c. bridges 
gıik,iıg, = 
(11g) 1k, 14(g 4k) are in BC. 
d. bridging 
gek, leg, 
(re g)ek,le(g e k) are in BC. 

2. (gek) € BC iff dom(k) = diff (dom(g)), 
(1¢g) BC iff cod(!) = diff (cod(g)), 
(legek) € BC iff (gk), (leg)e BC. 

3. (gık)e BC iff diff (dom(k)) = dom(g), 
(I 1 g) € BC iff diff (cod(i)) = coa (9), 
(11g Lk) EBC iff (g 1 k), (I1 9) EBC. 

Bridging 

Assoziativity : 

If k, g, IEBC, then(keg)el=ke(gel), 

Bridging : 

bridging, ra” het (w,, a, ) ehom (a, w, ) «het [os a) — het (o, a, ). 

Bridge 

Assoziativity i 

If k, g, 1€ BC, then (k 1 g) 1 l=k ı (g 1), 

Bridge : 

bridge, u.) s het(o,, a,) l hom(«,, o, ) l het (w,, a) => het (o, a,). 

Bridges vs. Bridging vs. Jumping 

(! igi k)*(legek)2(I Ik), 

("ıg*r)&(1eg 1k) 2 (tk), 

(leg ık)&(tıgek)&(lk). 

diff (1) = (e), (1) = diff (e). 
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4. Bigraphs in diamond webs 


Instead of labelling transitions of the behavioural calculus, the whole sys- 
tem of bigraphs could be labeled (disseminated), i.e., distributed and medi- 
ated. Reflectionality between disseminated bigraphs, then might be realised by 
the “double-character” of diamonds. The possibility to disseminate bigraphs 
would open up a chiastic chain of connectivity and locality graphs, of stat- 
ics and dynamics, as a new play of interactionality/reflectionality between 
bigraphical systems. 


4.1 Disseminated Diamonds 


Salt ] [Cat‘ ] 
[cat] [Cat ]/ [Sar] [sale ] 
pea I 
[Sar] /[sare ] [cat] / [cat* | 
NS FEN 
[cat] [cat i / [salt] Salt” | 
BR ENZ i 
[Sart] [Cat] 


Diamonds, in this possible dissemination, are mapped as categories and 
saltatories with their dualities. 

Mediation between diamonds happens horizontally, by complementarity 
and accretion from dual-categories to saltatories. And vertically, by duality 
and iteration from one diamond to another diamond of the grid.*° 


4.2 Towards a diamond web of bigraphs 


In this setting we would have to introduce first the dual theory of bigraphs, 
which are themselves incorporating the dual structure of topography and 
connectivity. The more intriguing step would be to develop the complemen- 
tary system to bigraphs and its duality, placed in saltatories. Both together 
are building the diamond of bigraphs, which then could be disseminated to 
model and design interactionality and reflectionality in a polycontextural sys- 
tem of interaction including the chiasm of global and local situations. Such 
a diamond web would not be restricted to informatic and physical global 
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interactions like bigraphs but would be open to offer a framework for knowl- 
edge related semantic and pragmatic aspects of pluriversal computation and 
communication. Dissemination of diamonds might offer a scheme for a dis- 
tribution and mediation of the orthogonality of connectivity and locality in 
bigraphs, which are themselves thematised as dualities.*! 

From a more futuristic vision, also with not much theory, Hai Zhuge 
(Beijing) develops the idea and sketches some steps towards a methodology of 
a knowledge grid, which is to “foster worldwide knowledge creation, evolution, 
inheritance, and sharing in a world of humans, roles and machines. ”*? 
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a Kä & 5 a Fa ra 
LESS Sf Ef 
2 ¢ & s & $ Es Ka S 
EEE 
Á ar Attunement 
< 
s S 
s & s$ 
é oe So-called new media art, occasionally referred to as “science art”, undoubt- 
s D 
P edly creates paradoxes. It has repeatedly been pointed out that one of the 
kad 
i main causes for these paradoxes lies in the “ontologische Indifferenz”! of new 
K media art as indicated by the oxymoron “science art”. Is it science or art, both 
oe or neither of them? If a scientist (who without doubt is socialised differently 
< 
r from an artist) gets caught up in the maelstrom of media art and is keen on 
s$ the new possibilities it offers, then she or he is at risk of being squelched 
F under the wheel of the discourse as a result of the prevalent ambiguities. The 
$ s$ convergence of art and science stimulates a kind of an immune system whose 
$ antibodies can be called destruction or deconstruction. 
oo Interactive media art is systemic per se. With regard to this cybernetisa- 
NZ 
ca S tion of art and science, a growing “affirmative negation logic” can be observed. 
u 
+“ System criticism becomes constitutive for knowledge and social systems that 
af P are criticised. Any escape seems difficult. On the one hand, art history pre- 
À PG dominantly wants to see the overcoming of the avant-garde’s definition of art 
s S 
Fa $ $ as general criticism. If, on the other hand, an artist addresses a scientific 
S E g 
e oe issue without explicitly criticising or satirising, then she or he is reproached 
ra for having an anachronistic recourse to a romantic concept of nature. Science 
in turn takes up avant-garde art’s self-referentiality as essential for its own 
concerns and attempts to develop synergetic models of creativity from it that, 
$ 
£ in a sense, are composed of two antagonistic poles.? This indicates exactly 
> & £ 
oe & how the negative logic leads to the absorption of art into the system. 
2 Annette Hünnekens and Claudia Giannetti are among the first who 
9 
E worked on a summary of hypotheses and a derivation of a theory for inter- 
S active art, respectively.” Whilst Hünnekens discusses different artistic and 
= 
S S theoretical positions and explains the underlying paradigms, Giannetti out- 
om 
S lines an endo-aesthetics as part of digital aesthetics, which itself can be con- 
S 5 
S Pa ceived of as paradigmatic. The endo-aesthetic concept directly follows Otto 
: E. Rössler’s endophysics, which is a natural scientific theory that radically 
$ ; : REES : f RASE 
s renounces the subject-object distinction.* Endophysics and its proximity to 
E interactive art is controversially debated and serves as an instance of how 
$ transgressions between art and science can create enormous tensions in the 
E za 
S 1 Hünnekens refers to a lack of distinction between art and science and speaks in 
2 r this context of a “crisis of ontology” (Hünnekens 1997, p. 16). Mersch and Ott discuss the 
Â 2 “historical differences and indifferences between art and science” (Historische Differenzen 
Pd und Indifferenzen zwischen Ktinsten und Wissenschaften) (Mersch/Ott 2007, p. 9). I 
$ have chosen the term “ontological indifference” in order to refer to Heidegger’s notion of 
& S “ontological difference”. 
Pa 2 Tröndle 2007, Tschacher/Tröndle 2005 
3 Hiinnekens 1997, Giannetti 2004 
A 4 4 Réssler 1992 
s 
S $ Ka ro e 
Fa S Es E 


NZ 
F S 5 
P & Pà & P 
s 
4 E E 
D S 


natural sciences, too. As a rule, however, border-crossers are barely recog- 
nised in the natural sciences. “Artists-in-Labs” or similarly called programs 
have no corresponding “Scientists-in-Studios” programs or the like. Here the 
meta-level of reflexion is obviously attributed to art. 

Within the science enterprise there does not seem to exist a larger need 
to reflect on the interrelation of art and science and thus about paradoxes, 
except for rhetorical reasons, as in: “we are also creative, so somewhat artis- 
tic after all.” Often, a rudimentary understanding of aesthetics settles the 
matter, reducing art to the production of the “beautiful”, since all the same 
“beauty” underlies scientific motives too. A second component of the scien- 
tific referring to art is its attempt to explain art as an emergent phenomenon 
within the scope of complexity theory, supporting its reduction to the “beauti- 
ful.” Finally, art is seen in its functional role as a generator of creativity.® 

In the following, starting out from existing perceptions amongst scientists 
of what the role of art is for science, I will try and work out a hypothesis on a 
paradox of interactivity based on Heidegger’s use of the concept of reification 
(Verdinglichung).’ I will also pick up on some philosophical positions which 
attest the avant-garde stream to have contributed to their own absorption 
by science. According to Axel Honneth’s interpretation, which comes close to 
Heidegger’s application that I adopt here, Verdinglichung is a failure of Being 
(Seinsverfehlung).® The ontological notion that “the world is a differential equa- 
tion” is rarely expressed in such an explicit way, however, de facto implied in 
the scientific practice to a large extent. This can be considered as only one 
possible manifestation of Verdinglichung. The represented gets equated with 
the representation and, therefore, deprived of its existential quality. 

Hünnekens has already pointed towards the difference which is reduced 
or missing due to interaction and by which some theoreticians even conclude 
that interactive installations cannot be art. I myself interpret this missing 
difference as an increase of Verdinglichung. This would abrogate Heidegger’s 
distinction between science, that shows a tendency toward Verdinglichung, 
and art, which possesses a potential to wriggle out of Verdinglichung. This 
trend presents a certain consequence of the historical evolution of art. Above 
all, media art’s proximity to technology and to topics of the natural and social 
sciences has convincingly been associated with a culmination of two tenden- 
cies that have been laid out by the avant-garde: its definition as a pure nega- 
tivity as well as the integration of life into art. The result of this integration 
is precisely the indifference which should be conceived as a chance for the 
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7 Objectification or reification are possible translations. I nevertheless prefer to keep 
the German expression in order to avoid blurring the meaning (Jahraus 2004). 
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emergence of new cultural cornerstones from the perfect mixture, rather than 
as an occasion for polemics or perplexity. 

The scientific tendency towards Verdinglichung — although not introduced 
as an essential concept in a Marxist context until Georg Lukacs’ — was criti- 
cally addressed by philosophers of life in particular.!° Simply put, science 
reduces nature to its essences in terms of measurement or observational 
values. The technical applications of Shannon’s information concept sharp- 
ened this tendency. In my judgement, Shannon’s mathematical definition of 
information has commonly been generalised in an ill-considered way. One 
can speak of “datafication” or “cybernetisation” of life. As the complement of 
the essences!!, the existences, can hardly be specified, the philosophy of life 
was often testified to have had a mystic character. Those scientists (e.g. Ilya 
Prigogine), operating on the edge of that mysticism, who want to dispense with 
the narrow systemic corset yet define themselves exact scientists rather than 
vitalists, are at a loss for explanations.!? This is mainly due to science’s lack 
of a conception of time. Introducing the concept of the “existential”, Heidegger 
created a philosophical framework within which that which (almost) defies 
discourse can nonetheless be thematised in a performative way. Jahraus 
speaks of an “auto-performance”!* with respect to Heidegger’s philosophy 
and relates it to the hermeneutic circle. A result of the effort not to fully 
detach science from Being is the incorporation of artistic degrees of freedom 
into the system theoretical approach. Exactly this, however, contributes to a 
kind of systemic conditioning of art'* and to the aforementioned indifference. 
The path of a performative science proposed here from the scientific perspec- 
tive, attempts to abandon the representationalist reduction of art and rather 
highlight its performative power. What originates as a result of the adoption 
of performative concepts is not per se conceptualised as art, but merely an 
attempt, in the fashion of art, not to disregard existence (Dass-sein, cf. foot- 
note 4). Contemporary art is no longer dominated by the paradigm of the 
avant-garde. According to my hypothesis that an increase of Verdinglichung 
results from the application of the avant-garde’s conception of art especially 
with respect to its relation to science, this renunciation is welcome. However, 
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10 Amongst them, Henri Bergson (1911, 1948) counts to the most prominent because 
the importance of his work for contemporary system theory has repeatedly been stressed 
by Nobel prize winner Ilya Prigogine (1985). 

11 “Essence” is a vexed issue. Scientists speak of grasping the “essence” when they 
attribute measures to something. Some philosophical streams regard “existence” as the 
“essence” of Being. I follow Heidegger who discriminates being in “Was-sein” and “Dass- 
sein”, i.e. “Wesen” and “Existenz” or essentia and existentia (Jahraus 2004, p. 193). In 
other words, “existence” is what fails when taking measures. 
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my provocative supposition is that the subversive avant-garde paradigm is 
not obsolete when being incorporated into scientific methodology. 


Retroactive Systems 


Hünnekens mentions, in passing, Hans-Peter Schwarz’ suggestion of call- 
ing the new stream, “retroactive art”.! From a system-theoretical point of view 
the disappearance of this notion is regrettable. Systems are called “retroac- 
tive” if they include components that are capable of modelling those systems 
in order to enable the derivation of an intervention strategy from a simulation 
of that model. 

Retroactive systems are thus subject to change exactly because we model 
them. They are dealt with in psychology, ecology, economics and sociology, 
to name but a few. In the course of their theoretical pervasion such sys- 
tems create problems of self-referentiality that are the subject of both sec- 
ond order cybernetics and endophysics. It is retroactive systems in particular 
that render performative methods almost inevitable and provide an excellent 
reason to make methodological borrowings from retroactive arts. Being a dis- 
ciple of Otto Rössler, my own transgression has been evoked through endo- 
physics as well. The subversive idea of the brain’s thermic noise entailing an 
uncertainty that projects onto the outer world is only one instance of a figure 
of thought that can be encountered in media art. For many years now, the 
media artist Bill Seaman has drawn explicitly on endophysics and has been 
cooperating with Otto Réssler.'® 


Remarks on Verdinglichung 


Interactive media art draws per se on a cybernetic world-view. Recipients’ 
measurable state variables are used to control the rest of the “machinic 
eigenworlds.” Such an art is at a risk of increasing the degree of cybernetic 
Verdinglichung. Paradoxically, an interactive media installation offers science 
the possibility of reducing Verdinglichung. The integration of life into art or 
vice versa leads to a balancing act between contingency and habituation, 
between performativity and repetition. 

Verdinglichung is seen here as a gradual property.” I assume that an 
absolute absence of Verdinglichung does not exist but can only be approxi- 
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mated. Heidegger operates with the notions of “present-at-hand (vorhanden; 
Vorhandenheit)” and “ready-at-hand (zuhanden; Zuhandenheit)” in order to 
emphasise the necessary detachment from ordinariness with respect to the 
appraisal of art.'® Interactive art is not only present-at-hand but also ready- 
at-hand - in accordance with the avant-garde’s demand to integrate life, but 
at the same time increasing the inclination towards Verdinglichung. I identify 
an entire Verdinglichung with the complete abdication of man’s (capability of 
his) freedom of choice towards the machine. These are cases where - to put 
it crudely — the human is condemned to being a machine that merely nods 
things through. 


EyeVisionBot 


By means of “EyeVisionBot” (Fig. 1), an interface for image search, the 
above introduced line of thought can be demonstrated in an exemplary way.’ 
From a technical point of view the device consists of an eye-tracking unit, a 
database (potentially the www), a visual display and several computers that 
host the control soft- 
ware. The latter controls 
and analyses the gaze 
tracking, accesses the 
database, and steers the 
visual output. Simply 
put, the eye-tracking 
device detects the view- 
ing direction of the user. 
With this it is possible, 
given a display oftwenty- 
five images arranged in 
a5x 5 matrix, to deter- 


mine which images are Fig. 1. EyeVisionBot 
EEE 


looked at and for how 
long (Fig. 2a-b). Those images being momentarily looked at become slightly 
magnified in order to give visual feedback. After a certain time, five times five 
new images are retrieved from the database and shown on the display. This 


EEG 
18 Jahraus (2004, p.79) summarises Heidegger’s thoughts on art as following: 
“Umgekehrt aber sieht er in der Kunst einen ästhetischen Ausdruck dessen, was die 
Technik gerade vergessen macht: die Teilhabe am Sein. [...] Während Heidegger an der 
Technik Verdinglichungstendenz metaphysischen Ausmaßes herausarbeitet, soll die 
Auseinandersetzung mit der Kunst gerade dazu dienen, diese Verdinglichungstendenz 
zu überwinden.” 
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time, instead of a random selection (as was the case with the initial access), 
the detected gazing-durations are used to pre-estimate the most desired 
categories, which are then preferentially accessed. A category is defined by 
potential classifications applied to the database and structural similarities of 
the images. Given a non-classified database, the search can be made on the 
basis of structural relations alone. The structural comparison is conducted 
with the open source software GIFT (GNUImage Finding Tool). 

In a museum installation (Fig. 1) we used the image database of the “media 
art net” project.” If, for example, out of the first 25 presented images, a pho- 
tograph of a media installation is looked at for a notably long time, then the 
user may be interested in this style of installation, have a particular interest 
in the corresponding artist or may be looking for another image that resem- 
bles the one she or he singled out. Several models that can be used to control 
the subsequent search are possible. The assumption underlying those mod- 
els is that the momentarily presented 25 images’ competition for the user’s 
attention would eventually lead to a distribution of gazing durations which 
reflects the user’s priority distribution with regard to the corresponding cat- 
egories. A universal algorithm for modeling and simulation of such subjective 
weighting in decision-making processes is based on Bayesian inference. It 
involves continuous re-weighting of possible hypotheses on the basis of given 
observations. Since it is usual, during each ‘turn’, for more than one image 
to be regarded for different periods of time, under certain conditions one can 
quite robustly estimate the desired categories within one cycle. The quota 
of the subsequent turn’s categories is calculated proportionally to the gaz- 
ing durations. A precondition for an efficient adaptation is a definitely fixed 
task such as, for example, the search for a specific image whose appearance 
is roughly memorised but for which neither the artist nor the style can be 
recalled. The structural resemblance then leads quickly to success. 

We originally proceeded on the assumption that the interface, along with 
further developed software, would constitute a creative tool for establishing 
dynamic user-generated database ontologies. However, it became evident that 
user modeling merely maps the set of prejudices onto itself. In other words, 
the interface as originally conceptualised only functions in a satisfactory way 
if a relatively precise aim governs the search. 

Assuming a perfect adaptation of the system to the user’s preferred cat- 
egoryin interaction with the user, the system will always stay in this category 
and without a special interference - like randomly adding images out of arbi- 
trary image categories — a change to another category will be impossible. For 
considerably more complex decision-making processes like in medical diag- 
nostics for which the user’s preferences are automatically anticipated by the 
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Fig. 2a. First array of presented images. The momentarily watched image is 
slightly magnified. 
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Fig. 2b. Second array of presented images indicating structural and taxo- 
nomic similarity. 
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algorithmic system, suspicions could be raised that a kind of “self-imposed 
nonage” of the user sets in as a result of taking pleasure in the release of cog- 
nition. If, in addition, the decision-making process is one of high responsibil- 
ity?’ then - according to my suspicion - the people will surrender themselves 
to the “objective” algorithmic decision for convenience. In other words, what 
was intended as backing for decision-making ends up in transferring the 
decision along with the “responsibility” to the algorithm. Human error turns 
into technical failure. This would then be high-degree Verdinglichung. 

Therefore, with regard to creative applications it seems to make more 
sense to investigate different deviations from the optimal user model. However, 
there are no rules as to how to achieve this. Only a causally open system 
can be creatively utilised. In my opinion, however, it is possible to generate 
an understanding of the mechanisms behind the user’s handling by means 
of performatively approached “museum field studies” without being able to 
directly translate these mechanisms into a definite mathematical model. 
Furthermore, a museum installation functions as a critical interface.”? One 
might, for example, employ EyeVisionBot to scrutinise one’s own habits by 
uncovering the normally invisible algorithmic decision processes. The lack of 
a significance value certifying the tool to be more efficient than other meth- 
ods for particular tasks so far hindered any publication in a professional 
journal. Therefore, the museum seems to be both the genius loci for perfor- 
mative scientific studies and a means for its publications, something which 
EyeVisionBot is intended to give an example of. A crucial point is the physical 
presence of the person whose cognition is to be understood partially. What is 
being modeled becomes part of the model. 

One could have the suspicion that due to inevitable user-modeling, inter- 
activity has a tendency toward Verdinglichung, but can at the same time avoid 
complete Verdinglichung as long as the users are allowed to be physically 
involved. 


Remarks on the Bayesian Algorithm 


Until recently the aforementioned Bayesian algorithm was mainly used in 
medical diagnostics. This method for the estimation of the validity of hypoth- 
eses integrates current investigation results and prior knowledge. It is heavily 
criticised by some statisticians because the recourse to prior knowledge is 
equated with a dependency on prejudices. Subjective degrees of reliability are 
described by it. For only the last few years, however, the estimation of subjec- 
tive probabilities is emphasised as an advantage of the method, namely in 
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cases of anticipating human decisions. With it, subjective decision making 
processes can be quasi objectified. Software that we use on a daily basis 
contains context-sensitive, cognition-supporting algorithms that are identical 
or very similar to the Bayesian method. Examples are junk mail recognition 
or the Office assistant. Computer-based surveillance and control processes 
contain these algorithms. “Semantic” search engines are based upon related 
methods. Brain physiologists even claim that during our decision-making 
processes a Bayesian algorithm is executed.” Therefore, the method has 
already been compared with the hermeneutic circle** giving rise to the formu- 
lation of a “Bayesian Epistemology”.?° In my opinion, this is a categorical mis- 
take. With the aid of the Bayesian inference principle each decision-making 
process can be approximated, and this can be carried out more effectively 
the more it is based on invariants. New ideas, however, mostly originate from 
“irrational” decisions that the algorithm is unable to describe and cannot 
therefore simulate. The power of simulations lies in the description of station- 
ary systems rather than in contingent ones. 

Recently, a further step in objectification has been discussed in medi- 
cine as well as numerous other disciplines (Law, History, Economics, and 
others) under the heading of “evidence-based medicine”. There are database 
projects which allow for the retrieval of all accumulated previous decisions 
for the purpose of obtaining comprehensive and robust estimations of a priori 
probabilities. Occasionally, for reasons of objectivity, relinquishing the deci- 
sions based on the database content and its algorithmic evaluation to the 
algorithm itself becomes necessary. In this way, artificial intelligence enters a 
causally closed sphere, thereby degrading human decision-making, in much 
the same way as an epiphenomenon, to a nodding-through farce. The model- 
ling of retroactive systems in such a way (that allows for an anticipation of 
decisions respective of the activities of agents so efficiently that the latter 
readily accept the results), would mean a high degree of Verdinglichung. In 
the following, I wish to argue that with respect to works that are motivated by 
a putative emancipation of society, media art is at risk of co-designing such 
an “evidence-based” society. 
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Integration of Avant-garde into the System 


As mentioned above, retroactive art (science art) and the convergence 
of cultures that is generally discussed under the label of “Art & Science” is 
considered a logical consequence of the avant-garde movement. Boris Groys 
(2005) elaborates on this: 


Since the 1970s we have been living and functioning in a post-revolution- 
ary system of art. According to G.W.H. Hegel (1770-1851), all post-revolu- 
tionary societies are characterised by the fact that they prescribe rational 
goals, procedures and strategies to their members, and demand explana- 
tions, justifications and precise plans from them. It is obvious that our 
present art system functions precisely according to these rules. The claim 
of a single artist that his or her work is an unpredictable, creative act, 
seems obsolete, and is not taken seriously by today’s art world. [...] it was 
precisely the radicalisation of the notion of creativity by the revolutionary 
avant-garde that has historically led to its integration into the ‘system’. 
The avant-garde art saw itself as the embodiment of the pure negativity, 
as the medium of destruction and annulment of all traditional, mimetic, 
naturalistic art forms. 


The basic statement of “integration into the system” is affirmed by Dieter 
Mersch and Michaela Ott (2007) as well as Gerhard Gamm (2007). The afore- 
mentioned authors emphasise the role of cybernetics in this context. A sys- 
tem theory that pretends to include epistemological processes in its models 
and simulations almost necessarily presents an attractor for artists who have 
always been endophysicists in their self-conception long before the notion 
of endophysics was coined. In the year 2003, Lutz Dammbeck (2007) who 
is an artist-scientist and hence, like nearly everyone, a theoretician too, — 
with his documentary “Das Netz” (The Net), began to discuss the role of art 
within the cybernetic world conception on a meta level. In a recent article 
entitled “Re-Reeducation or: Art and Conditioning” he speaks alternately of 
a “digital dictatorship” resp. a “systemic dictatorship.” He repeats the posi- 
tion (already mentioned a number of times) that the avant-garde contributed 
to its own absorption into the system through its categorical system criti- 
cism. Dammbeck conceives the assimilation of art into the system in such an 
extensive way that having read his article one has to wonder whether there 
was any art after World War Two that was not engrossed in the system or 
found its legitimation exclusively from the system. 

Dammbeck fears a global brainwashing, and Pavlovian conditioning in 
which artists only have to play the “criticising class clown”. He says that it 
can be clearly seen 


that the idea of an ‘outside’ from which the ‘inside’ can be changed is naive 
in the face of patterns and structures designed by cybernetics and system 
theory, because each point at the periphery is at the same time the center 
and an ‘outside’ no longer exists. And we also know: the mere thought of 
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a possible change produces an energy that can be used by the system in 
the same way as every attack or perturbation as an intake of energy for 
further perfectioning. (...) Therefore, it would be meaningless to take action 
against it, since each critique not only preserves the system’s life but even 
strengthens it. Metaphorically speaking: Those who touch the machine are 
already part of it and its codes. 


In other words, after the cybernetic conception of nature, a systemic role 
was assigned to everything. I do not regard it as impossible that we are deal- 
ing with a kind of brainwashing paranoia which was called “cybernetic irony” 
by Peter Sloterdijk in a conference on the film “Matrix” (Sloterdijk 2000). What 
is noteworthy in this context is the video installation entitled Psych|OS, 
belonging to the distinguished actionist collective “Übermorgen”. The entirely 
confusing recordings were made by one of the members of the actionist group, 
Hans Bernhard, during his stay in a psychiatric hospital due to a serious 
psychosis. One should know that Ubermorgen belong to the most effective 
system critics. Hans Bernhard, for whom the quarrel with “the net” is a kind 
of a self-therapy, writes about himself and “the net”:?° 


Hans Bernhard’s neuronal networks are connected to the global network, 
and his mental illness - the bipolar affective disorder that in March 2002 
sent him to a mental hospital - is the network’s illness. The video called 
Psych | OS (2005) sums up this experience, in which those two levels - dig- 
ital and real, bio & tech, nervous system and operative system — merge. 
This nervous system, infected by the hi-tech, needs a treatment, and the 
hi-tech society prescribes its remedies, bio-chemical ‘agents’ which control 
the internal information flow. [...] The Psych |OS Generator (2006) is the 
literal application of this kind of control: a piece of software that asks the 
user about the symptoms of her disease and provides her with a remedy, in 
the form of a ‘forged original’ medical prescription. 


Viral Dynamics 


Within the area of “street-art” Julia Reinecke affirms the ontological 
ambiguity as a consequence of the avant-garde.” Here, it is the indifference 
between art and commerce. The relation this bears to the topic discussed is 
closer than it might be expected. 

Street-art is a form of actionism and in some respect is comparable with 
hacktivism, i.e. Internet actionism. Most street-art activists do not call them- 
selves artists, yet locate themselves within the tradition of situationism and 
other streams of avant-garde art which dedicated themselves to the integra- 
tion of life into art. Street-art explicitly locates itself between system critique 
and system conformity. This leads to a continuous innovation with respect 
to commercial trends, but this innovation is itself subsumed in commercial 


0 zz 


26 Übermorgen 2005 
27 Reinecke 2007 


153 


trends once again at a tearing speed. Interestingly, the activities are often 
fully detached from content. Similar to situationism activities are undertaken 
purely for their subversive or provocative impact. The whole art of action- 
ism consists only in the strategy. Unsurprisingly, the relatively young com- 
mercial movement called Guerilla marketing evolved directly from street-art 
whose main concern is attention economy, and where product information is 
pushed to the subliminal border. Guerilla marketing has recently re-imple- 
mented itself into street-art as Guerilla art, where quite frequently the artistic 
content is crowded out to the periphery in an analogous way. Many activities 
in Guerilla art and Guerilla marketing are conceptionally indistinguishable. 
For instance, within the “go public” actionism of Michael Bielicky there is, in 
the first instance, no talk of the content of the art emerging in a Guerilla-like 
way in public space.?8 

The mechanisms of the propagation of “signifiers without signified” that 
underlie Guerilla marketing led to the related concept of viral marketing.” 
This concept assumes the viability of modelling word-of-mouth propaganda as 
epidemiological dynamics. In this way, two previously autonomous currents 
in dynamic models of cultural evolution converge: memetics and marketing. 
Memetics claims to generalise Darwinian theory of evolution and to be capa- 
ble of describing cultural evolution by virtue of comparable mechanisms.*° 
Corresponding to genes, the basic units of culture are memes, which spread 
and survive according to the laws of selection of the fittest. The concrete 
propagation dynamics is equivalent in its form to the proliferation of viruses, 
giving rise to the name “viral marketing”. Similar to genetic engineering, the 
concept of viral marketing assumes that specific phenotypes can be designed. 
This is particularly easy to do on the Internet, because easily accessible infor- 
mation (tags, newsfeeds, access statistics, memes detectable via data min- 
ing) on bloggers’ habitus in the subcultural field, the “blogosphere”, can be 
used to monitor, model and design this part of society. Here, epidemiological 
dynamics are coupled with graph theoretical models from network theory. 
By now, it is possible in some simple cases to calculate optimal conditions 
for meme propagation. Great efforts are being made to improve this analy- 
sis of structures and life-cycles of memes by employing pseudo-hermeneutic 
Bayesian statistics. 

Similar to tags and stencils in street-art, viral marketing (with the col- 
laboration of artists) inoculates the virtual world of the Internet with memes 
(videos, flash animations, games, etc.) that tout for attention. The strategy 
of viral marketing makes itself the subject of discussion, and so it is, to an 
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amazingly high degree, effectively self-promotioning. This is a typical “line of 
argument” corresponding to “cybernetic irony.” 

One noteworthy system is the monitoring system “Blogviz”*! with which 
the flow of memes in the blogosphere can be monitored. The correspond- 
ing Master’s Thesis contains a detailed chapter on previous artistic achieve- 
ments. As a particular example, I'd like to mention the prizewinning instal- 
lation “Listening Post” by Mark Hansen and Ben Rubin. The following can 
be read online on this installation: “Listening Post is an art installation that 
culls text fragments in real time from thousands of unrestricted Internet chat 
rooms, bulletin boards and other public forums. The texts are read (or sung) 
by a voice synthesizer, and simultaneously displayed across a suspended grid 
of more than two hundred small electronic screens.” This work is mentioned 
in several other publications on memetics (which cannot be listed here) as an 
important ground-preparing work. As an example it may suffice to show how 
memetic research and avant-garde art and, for that matter, “datafication” and 
“cybernetisation” of Being are interwoven, especially since this art is not only 
the subject of memetic modeling, but also, as in the case of “Listening Post”, 
develops memetic models by itself and designs them in a sensual way. 


Remarks on Cybernetic Irony 


The works of net activists often are of an intensely paradoxical form. In 
the case of “Amazon Noir”, Übermorgen hacked online bookseller Amazon's 
web presence in order to apply an efficient algorithm providing access to sam- 
ple pages that could be combined to complete books. Manually, this would 
take month or years. Amazon became aware of the hack and put Ubermorgen 
under pressure. Eventually, Ubermorgen sold the algorithm to Amazon and 
signed a non-disclosure agreement. The campaign is now exhibited in a 
purely symbolic way. Ubermorgen (U) was interviewed on this matter by the 
online journal Telepolis (T):°? 


U: Our projects are purely about experimenting: Amazon Noir is not a 
statement on copyright and even less an attack on the online trader. There 
is no specific goal behind it; the matter simply arose. I call it freestyle basic 
research. We build a setting and observe what happens sociologically, with 
relation to mediation, and technologically. We didn’t have a fixed plan of an 
outcome at the time. The sell-off arouse as a new solution, and so we opted 
for the agreement with Amazon. 

T: (...) there is no comment at all by Amazon on the alleged sale of the soft- 
ware. De facto the whole action might just as well not have happened and 
be merely merged. (...) Wouldn't this be the “next level” in media hacking: 
coverage of actions that have never happened at all? 
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Ü: Of course we already did such things and we experiment with it, but in 
our big projects like Google Will Eat Itself and Amazon Noir it is essential 
that the technological part functions (...). We are lazy-bones and it is tedi- 
ous having to make up everything! 


One can find an almost indefinite number of similarly absurd perform- 
ances, which leave the matter unclear as to whether they are viral market- 
ing activities, hacking or simply cybernetic irony. Incidentally, in the case of 
Dammbeck’s prognosis, the question of whether one should regard the spec- 
tre of a global cybernetic brainwashing as similarly being an artistic concept 
or as being a serious contribution to media theory, is virtually irrelevant with 
respect to its ironic impact. 


Back to Earth 


According to Mersch and Ott avant-garde, which to a great extent referred 
to Nietzsche, is a “reciprocal radicalization that accepted the challenge, not 
only to hold it’s own ground with respect to the sciences but to eventually 
imbibe them.”?? According to Nietzsche, Mersch and Ott elucidate, artistic 
practice is the “Ereignung von Ex-sistenz”™ itself. “Of a higher sense than any 
discourse, art literally reaches down to the abysms of Being.”, they further 
explicate. Actually one is inclined instead to diagnose art has having been 
merged into science, albeit in a manner that is just opposed to the “Enowning 
of Ex-sistence”. I therefore take Dammbeck’s diagnosis of the artist as a “criti- 
cal class clown” very seriously, but I do not share his pessimistic stance. It 
is now important not to repeat the mistakes made by the Frankfurt School, 
(who took a generally pessimist stance), but rather to undertake a critical but 
constructive approach. 

In my opinion, the necessity almost inevitably follows from this to pro- 
vide sciences with an understanding of the “existential” and the attempt to 
transfer the original avant-garde criteria to sciences, namely to integrate life, 
i.e. to create a causally open structure. It should be made possible for “agent 
causality” to be brought into the system. It obtains performativity, the enown- 
ing of existence is enabled. 

The lack of understanding among scientists usually created by recourses 
to existential philosophy such as “Enowning of Ex-sistence” usually came to 
light in the dispute between Ilya Prigogine and Jean Bricmont. Prigogine, who 
based his ideas on Henri Bergson’s process philosophy, was accused first by 
Bricmont (1995) and then additionally by Alan Sokal (Sokal/Bricmont 2001) 
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of abusing science. Not many attempts to change scientific methods and the 
conception of time such as that of Prigogine come from within natural sci- 
ence. Therefore, it makes sense to demand an intake of the concept of the 
existential into a science which is restricted to essentials. It is exactly this 
domain where participative, interactive, retroactive, performative or however 
described media installations abundantly endow a great deal of sense. Science 
receives corporeality, and reality becomes rehabilitated in a certain sense. I 
suggested that such a repertoire of methods, enriched with the existential, 
should be called “performative science”.°° From this perspective it follows that 
art is not reduced to a functionality in commission of science, indeed not to 
a functionality at all, and that the development of art in emancipation from 
science can even be advocated. However, it is mandatory to accept a new 
episteme that settles between traditional science and art. 
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1. 


The Fabrication of Life is probably one of the most ambitious, but at the 
same time one of the most controversial research fields within so-called con- 
verging technologies.! Whether Synthetic Biology”, Artificial Life’, or Bio Art* 
- all these fields are concerned either way with the manipulation or synthesis 
of living beings. 

It is well known, that organisms can be manipulated to some extent by 
altering their gene expression and this sort of research is already placed 
under quite heavy, political control. What is still less well known, and long 
since lurking behind the scenes, is artificial abiogenesis — the synthesis of 
life “from scratch”. This type of research — strangely enough - is done almost 
completely unnoticed by the general public. Bio Art, on the other hand, sel- 
dom bothers with all those technologies - it just “applies” them, from a more 
or less critical stance. 

A crucial question which soon comes to mind, though, is: what could 
“fabrication of life” actually mean? Of course - the less precise the definitions, 
the easier we could describe beings as “alive”. Thus, for the following, we 
assume that a “living being” is a “being-for-itself” (“subject”), that it is a being 
with its own proper world (“Umwelt / Innenwelt”), and that “life” is creative: 
that there is an emergence of “otherness”. This very process of emergence of 
something new — other — will be called “poiesis” or “in-formation” in the fol- 
lowing. And we mean explicitly “fabrication”: bringing life into existence (what 
has been referred to already as artificial abiogenesis). We do not talk about 
“manipulation”. 

Is such a project imaginable? Can we expect to become “life engineers” in 
the near future, building “Living Machines”?® 

A lot of people think it is. Some of the most advanced projects in this 
respect are those which are collected under the umbrella of the 6th Framework 
Program of the European Union, as there are SynthCells, PACE, Uniroma3, 
and Protocell Assembly for instance. All these projects are focussing on single 
cell organisms which, as general “building blocks” of more complex multi- 
cellular organisms, should, nonetheless, fit the aforementioned definitional 
“requirements”. 

Now, in order to discuss the problem of “life engineering” we want to 
exclude from the very beginning any framework which relies on some sort 
of “vital force” or a specific “bio-substance”; we don’t want to introduce any 
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type of transcendental dualism either, e.g. phenomenology. We want to stay 
as close as possible to contemporary scientific approaches. Having said that, 
however, we don't think that mainstream reductionism works. Reductionism’s 
underlying assumption is that living beings are machinic (in the sense of 
a mechanism). This assumption is a consequence of a much deeper reduc- 
tionist belief which is that biology has to be anchored within contemporary 
physics; and — usually - physics is here confined to Newtonian physics. This 
reductionism is not feasible for at least two reasons: first of all, sciences can- 
not treat “creativity” appropriately, and secondly, sciences cannot deal prop- 
erly with subjectivity. As it will turn out, the ultimate cause for this failure is 
a profound inability to theorise time. In physics, there seems to be no notion 
of time at all. And insofar as the “sciences” in general are desperately fixated 
on physics, they are suffering from the same problem, too. The humanities, 
on the other hand, are unable to bring their concepts of “subjective time” 
(and/or “social-historical time”) into any consistent contact with those “sci- 
ences”. As a result, the dialogue between sciences and humanities either 
stops, or creates amusingly bizarre “discourses” — most often, for example, in 
the neurosciences with its perennially recurring debates about the problem 
of, e.g., freewill.” 

One could thus get the impression that this very problem of fabrication 
of life indicates a paradox - it would be simply impassable. Even within biol- 
ogy, doubts exist about whether this reductionist strategy will work. Robert 
Rosen, e.g., argues, that physics “is inherently inadequate to accommodate 
the phenomena at the heart of biology. No amount of sophistication within 
these limitations can compensate for the limitations themselves.” In particu- 
lar, since Newtonian-style physics produces analytic knowledge (i.e. knowl- 
edge, how a system works), and this knowledge does “not entail how it is 
created”,? the problem of fabrication simply cannot be solved. 

Stuart Kauffman also considers the possibility, that contemporary phys- 
ics has to be changed to become appropriate for biology. His main concern 
is that for principle reasons we “cannot finitely pre-state the configuration 
space of a biosphere.”!° What he is speculating about is “glimmers ... of some- 
thing like a fourth law, a tendency for self-constructing biospheres to enlarge 
... the dimensionality of their adjacent possible.”!’ We can’t go into the details 


zz zz za 


6 Barbour 1999 


7 There is no doubt that the life sciences can produce results of some (limited) use, 
despite their dubious conceptual and methodological premises, e.g. in medicine with its 
often spectacular progress in diagnostic and therapeutic technologies, particularly in the 
neurosciences, see Hagner 2006. 


8 Rosen 2000, p. 256 
9 Rosen 2000, p. 258 
10 Kauffman 2000, p. 135 
11 Kauffman 2000, p. 244 


162 


here, but we just mention in passing that Kauffman expects a “new physics” 
— which eventually respects biological phenomena - to be a physics carrying 
on the ambitious efforts tackling the challenges of quantum gravity.'? 

Both Rosen and Kauffman, in a way, suggest that we have to change the 
underlying conceptual framework of mainstream reductionism, and, most 
important, have to explore alternate ontologies, the main emphasis of which 
is on being-for-itself, creativity, and “becoming”. We can go even further. Why 
should we conceptualise nature in terms of a “state space”, introducing a 
“mind-body-problem” — just to “reduce” it to “materialism” or “idealism”? And 
why should we continue rendering the “ego” as a spectator of an external 
“world” — mirroring predictable trajectories and desperately trying to stay the 
course? There is no compelling reason at all to believe in these somewhat 
accidentally sedimented clichés, acting — at the best - as a common-sense 
“doxa”. 

We should be aware, though, that taking non-reductionist frameworks 
into consideration often means to be accused of “vitalism”, “speculative 
thinking”, et cetera — as we already mentioned above. Authors like Spinoza, 
Nietzsche, and Bergson are usually condemned as belonging to the “bas- 
tard line of philosophers” (Deleuze), and there is rarely a chance to engage 
mainstream “scientists” in a profound debate.'* An exception might be A. 
N. Whitehead, whose “process philosophy” happens to be discussed as an 
alternate ontological framework for quantum physics.'* But Whitehead is still 
seen as quite an esoteric thinker. 

Among the (maybe) less suspicious philosophical authors who are deeply 
concerned with the problem of “creativity” Castoriadis comes to mind. A read- 
ing of Castoriadis seems to be rewarding from at least two points of view. As is 
well-known, he advocates the crucial role of “radical imagination” in human 
subjectivity. And, additionally, he develops an ontology of the “magma”, which 
— as will hopefully be shown in this paper — allows a rethinking of “creativity” 
in such a way that it sheds a new, interesting light on “fabrication of life”. 
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2. 


Castoriadis’ philosophy, and especially his ontology, remained unfinished. 
It always had a strong momentum, most notably in his late writings,'® which, 
as I would suggest, can even be read as constituting first “building blocks” for 
a proprietary process philosophy. 

From its very beginning his philosophy is centreed around the concept 
of autonomy. Autonomy is the result of a process of self-constitution or self- 
creation, both on the individual and the social-historical level. We will only 
briefly touch on this topic here, although it opens many opportunities for 
criticising current approaches in neuro-, brain and cognitive sciences, first 
and foremost in neuropsychoanalysis. 

The genealogy of autonomy is driven by imagination, “radical” as well 
as social-historical. Imagination as radical turns out to be the “differentia 
specifica” of human beings, compared with animals in general, which exhibit 
imagination in a functional organic context only. Humans, in contrast, have 
their imaginative capacities detached from any functionalisation — imagina- 
tion becomes free floating: radical. During individuation — a process of psychi- 
cal “sense- or meaning-making” — radical imagination evolves into both an 
“interior” (psychical) and an “exterior” (social-historical) equilibrium of rep- 
resentational pleasure, implying a compossible coupling with the underlying 
organic functions as well as a proper embedding into social-historical imagery. 
This “individuation” might fail - in the worst case resulting in psychosis.” 

Thus, subjectivity of humans is anything but a fixed, rationally behav- 
ing “agency”; this might be the case, according to Castoriadis, with animals 
and their “hard-wired” (yet still representational!) pleasure, entwined with 
organic functionalities. Whereas with human beings, it’s just the opposite: 
subjectivity is the felicitous result of an emergent creative process of radical 
imagination, susceptible to failure, but also open to revolutionise the world 
by creating “other” imageries. 

The first lesson we can learn from Castoriadis, therefore, concerns the 
processual character of subjectivity, thwarting the mainstream caricature of 
rational agency. If at all, the latter turns out to be a (cynical) zoomorphism, 
turning the creative capacities of human imagination into pre-determined 
sensor-actor-circuits, receptible for computational or dynamicist models. 
And Castoriadis even gives us arguments against a naive adaption of the 
Freudian project. Whereas the latter confines psychoanalysis to the private 
context of the doctor’s couch, Castoriadis emphasises the role of the social- 
historical mediation of imagination. That which happens at the border of 
16 Adams 2003 ~~ 
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“Unconscious” and “Consciousness” is not just the “personal” fluctuation of 
imageries; both its genealogy and its actual virulence are deeply entangled 
with the exterior, social-historical. The subject is always “a bastard construct, 
combining in various proportions elements of the psyche, of the social-histor- 
ically instituted understanding and reason, and of the self-reflecting activity 
of the social individual at a certain stage of history.”'® 

As mentioned already, Castoriadis’ philosophy needs to be thought of “in 
terms of a shift from a regional ontology of the social-historical towards a 
transregional ontology of physis”, as Suzi Adams puts it.!? What becomes 
the main concern throughout Castoriadis’ later writings is the logical as well 
as ontological difference between determinacy and indeterminacy. In order 
to unfold the complex interplay of both these “dimensions” of being, we first 
have to become aware of the stratified character of his “transregional” ontol- 
ogy. Physis subsists as a dynamic multiplicity of (strata of) being, which 
“is an irreducible, primary datum.” As such multiplicity “formally entails 
unity”.?! Without unity, multiplicity would cease to be multiplicity, and would 
become an “in itself dispersed and disconnected Infrachaos”.?? Now, there are 
actually two ways how multiplicity exists — as difference, and as otherness. 
This distinction belongs definitely to the core of Castoriadis’ conceptual appa- 
ratus, and it is immediately entangled with determinacy and indeterminacy, 
respectively. 

Let us start with an example: a square is different from a rectangle, but 
Kafka’s “The Castle” is not different from the Rolling Stone’s “Satisfaction” — 
they are other. According to Castoriadis, two forms are different “if there is 
a set of determinate transformations (‘laws’) allowing the deduction or pro- 
duction of this form.””® “Determination” has to be taken in its most general 
reading, as being an identitary element of an ensemble - i.e. set-theoretically. 
Because of its overarching importance Castoriadis coined a new term for this 
“ensemblistic-identitary” logic: ensidic. This logic is “hard-wired” into our lan- 
guage; it is the basis for all mathematical constructions, and is the underlying 
logic of our sciences.” Theorising along ensidic lines results in a construction 
of hierarchies of sets, equipped with relations and rules of deduction. Ensidic 
thinking “spatialises” multiplicities insofar as it constructs unities by identi- 
Jying elements and collecting them as an ensemble. It neglects any intrinsic 
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characteristics and figures these elements simultaneously, yet coexisting as 
different ones just by external organisation. 

There is no depreciation of ensidic descriptions (in contrast, e.g. with 
Heidegger’s “Vorhandenheit”), rather the opposite is the case — they constitute 
the dominant form of our world representation. Because the “first natural 
stratum” itself allows for ensidic constructions, social-historical imagination, 
individual humans are capable of instituting viable representations of their 
respective worlds. The problem arises if we exclusively turn our attention to 
ensidic narratives, reducing the multiplicity of being to simply a differential 
one. A world made up only out of differences wouldn’t change anymore, and 
nothing new would happen. All is determined, only differences exist: the rep- 
etition of the same. But the “new is not the unforeseeable, unpredictable, 
nor the undetermined.” The (unpredictable) next number in roulette, for 
instance, still remains “the trivial repetition of a form”, as does the undeter- 
mined, “sheer repetition of a given form” in quantum mechanics.? The “new” 
requires the indeterminate, the magma, which allows for the emergence of 
new determinations, of new laws; this “is the meaning of form - eidos.”?” 

How does this in-formation (poiesis), the emergence of the “other”, arise? 
We already mentioned the second way of how multiplicities exist: as other- 
ness. Otherness cannot occur out of ensidic space. Ensidic space only knows 
of differences, forms, where each form can be derived or produced from other 
forms, by determinate laws. No new forms emerge. Hence, we might consider 
time. New forms emerge in time, don’t they? — It depends. 

Castoriadis’ extensive analyses first show us why “creation”, the emer- 
gence of “otherness”, can’t be described by physics and related sciences. The 
reason is simple: they see “time” exclusively as ensidic time — social identitary 
time, which leans on the ensidic dimension of the first natural stratum.”® 
This, in turn, implies the spatialisation of time in the sciences, and results in 
the reduction of temporal multiplicities to differential ones. 

Therefore, time in general does not really help. We have to take into account 
the magmatic dimension of time. The emergence of forms (in-formation) is the 
ultimate character of time. The “before” and “after”, the irreversibility of poietic 
time, is “given through the scansion of creation and destruction.”? Poietic 
time forces a self-deployment of new forms in ensidic space and time as recep- 
tacles of the first natural stratum, where they become organised through sub- 
jective - both social-historical and individual - constructions. Forms as forms 
are not caused by something, in the sense of determinate necessary and 
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sufficient conditions; they emerge - given appropriate (innumerable, but only 
necessary) conditions. “The conditions allow the emergence of the form — but 
the converse is meaningless.”*° In-formation is ex nihilo, which does not mean 
in nihilo, or cum nihilo. The magma allows for in-formation, but it cannot be 
exhausted; the ensidic is indefinitely “dense” in the magmatic. 


3. 


As previously suggested, we could start reading the late Castoriadis as 
process philosophy. In particular, if we focus on his text “Time and Creation”,*! 
we will detect a clear prominence of time over space — the two “receptacles”. 
When Castoriadis asks whether there is “a possibility for an essential distinc- 
tion between time and space”*’, in the end he gives priority to time: without 
time there would be “no thing (nothing)’**. Nonetheless, time and space are 
intimately entangled for multiplicity exists both as difference and as other- 
ness, and “otherness entails difference”**. This, in turn, implies that every 
form - in order to be — has to be “identical to itself”, it has to persist for a 
while, qua pure repetition in ensidic time - differing with itself “only by being 
placed in a different (identitary) time”. Thus, every form has “necessarily 
an ensidic dimension”. And Castoriadis’ ontology establishes a clear priority: 
being is time. “The fullness of being is given — that is, simply is — only in and 
through the emergence of otherness which is solidary with time.”*° 

Finally, then, in-formation (or poiesis) — the “surging forth” of otherness as 
characteristic for being - forces the fragmentation and stratification of being. 
Qua self-deployment, being forces the proliferation of otherness, dispersing 
new forms both in poietic and ensidic space and time. As poietic receptacles, 
space and time ensure alterity; as ensidic, they establish the Being of being 
at all. Thus, the emergence of otherness, in-formation, does not contradict 
determinism; it rather contradicts “the paradoxical, if not absurd, idea of 
a homogeneous universal determinism that could reduce level or strata of 
being (and their corresponding laws) to a single ultimate and elementary 
level.”?” Creation ruptures the smoothness and continuity of being, it foils 
reductionism. 
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It might be worthwhile looking at contemporary sciences with respect to 
their concept of emergence. Emergence - as is well-known - is currently often 
seen asanew “weapon” in the hand ofreductionists. Teeth-gnashing, physicists 
are beginning to accept that there are “levels” or “strata” of being — probably 
forced by their very problems with thermodynamics and quantum mechan- 
ics. Yet ideally, these strata are communicated as hierarchically ordered, with 
a one-way determinism from the bottom up. For example, the (phenomeno- 
logical) variables of thermodynamics (like “temperature”), which constitute 
a “higher level” of description, are reduced to the movement of molecules 
— a “lower” descriptive level, more “fundamental”, and thus explaining the 
macroscopic phenomenon. This example illustrates exactly what Castoriadis 
complains of as “homogeneous universal determinism”. Admittedly, more 
advanced conceptualisations of “emergence” are “emerging”! — it might be 
Logic of Magmas”*®. 

Castoriadis tried several times to elucidate his concept of “magma”, the 
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promising to relate them with Castoriadis 


most detailed attempt probably is his paper “The Logic of Magmas and the 
Question of Autonomy”.*° I don’t want to comment on this paper here explic- 
itly; rather I would like to emphasise his reference to quantum mechanics. 
In this paper he mentions Mugur-Schachter, a physicist, talking about her 
reflections during theory-building in quantum mechanics. In the end, she 
finds herself within a “semantic mud”, and “it is only here, in this mud, and 
when we force our gaze to make out the moving forms, that we can perceive 
the contrasts between what is not done and what is partially done and thus 
initiate something anew.”*! Mugur-Schachter alludes to their problems with 
the vanished subject-object separation in quantum physics, and the difficul- 
ties of handling these problems semiotically. 

This reminds us - and that is probably the reason why Castoriadis quotes 
her - of his account of subjectivity. Again, in his paper “Time and Creation’”’, 
Castoriadis develops his concept of space and time, and how they relate to 
subject and object. Just remember: the world is socially constituted (via 
imaginary institutions), and it “appears as the deployment of two recepta- 
cles, social space and social time, filled with objects organized according to 
relations, etc., and vested with meaning.”** Receptacles appear to a subject. 
But they lean on the first natural stratum with “respect to [their] ensidic 
dimension”**. Every living being (being-for-itself) “know(s) ... at least some- 
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thing of the world.” This implies that the world “is knowable”; but it has to 
be “constructible” as well. The world “must contain the ... equivalent of 
an identitary dimension.”** We don’t have a chasm between “subject” and 
“object”, however - as is still the case in mainstream thinking, as long as 
it relies on Newtonian physics. It is rather a chiasmus (Merleau-Ponty) of 
subject and object; their respective parts of these constructions cannot be 
disentangled. Yet, our effort to separate them is not “... meaningless, on the 
contrary; but it is bound to be interminable.”? 

We might understand better now, why Castoriadis was seduced by quan- 
tum physics: the latter turns out to be the reference for our interaction with 
the world. The fundamental interactional pattern between subject and object 
is quantum mechanical, and not Newtonian. Our world is a world of “Zing!”*® 
Jean-Yves Girard is one of the first logicians who strongly emphasises that we 
have to stop imposing an ensidic logic (to use Castoriadis’ term) in theorising 
about nature (as so-called quantum logic does); rather we should take non- 
commutativity seriously, and create a new logic which picks up the insights 
of quantum physics, and develop a logic along the lines of the principal imbri- 
cation of observer and system.*?” One of the most fascinating results of this 
approach is the relativisation of set theory - it simply becomes “local”, a sub- 
jective “viewpoint” of an observer. If we recall Castoriadis’ attempt to describe 
the interplay between the magmatic and the ensidic, the emergence of “new” 
determinations during the interaction of being-for-itself and its proper world 
- maybe with Girard’s “Geometry of Interaction”? we have found a promising 
departure for the conceptualisation of the “everywhere dense” ensidic within 
the magmatic - in-formation. 


4. 


If we now turn back to our very question — How is fabrication of life possi- 
ble? — we should first stress the fact that Castoriadis’ philosophy/ontology in 
general seems to be an attractive, competitive framework for theorising about 
life and technology — from single cell organisms up to the attempts of the 
neurosciences to model human behaviour. Unlike Heidegger, or other philoso- 
phers in the phenomenological tradition — with their often exposed techno- 
phobic attitude — , Castoriadis has a brilliant background both in mathemat- 
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ics and in the sciences, which often allows him to avoid bizarre conclusions 
and misinterpretations. 

Secondly, there exists an explicit examination of Varela’s approaches 
to biology,®! focussing mainly on the concepts of autonomy and being-for- 
itself. The crucial question is whether we can think “the living being as a 
Jully ensemblistic-identitary automaton”; an automaton, that “has in itself 
the principles of its generation and corruption as well as of its alteration.”°? 
Castoriadis simply does not know. Yet he doubts that it could be possible, for 
the following reasons. Even ifit would be possible to create a complete ensidic 
description (and construction) of a dog - including an isomorphism between 
the dog’s own significations and constructions within its proper world, and 
the external ensidic description of the automaton - even then, this “artificial” 
dog wouldn’t be “new”, it would be just a replica of an already existing system, 
whether this is “in the head” of the engineer, or a natural “template”. And 
this, according to Castoriadis, seems to be implicit in ensidic logic: we would 
never have the reason, nor the criterion for fabricating the dog, if “the dog did 
not already exist.”°® 

We could go beyond Castoriadis (and Varela), and might consider implicit 
fabrication. This would generally imply the ensidic determination of a “param- 
eter space” — whether discrete or continuous. It would need a (determinate) 
“quality measure” as well as a (determinate) “procedure”, driving the system 
through the parameter space. Eventually, a being-for-itself might “evolve”. Two 
cases can be considered. First, the system “emerged”, it worked as intended, 
and nothing “extra” happened. This wouldn’t change the scenario at all, the 
same arguments as before would still be valid. In the second case, though, 
we could imagine that this being-for-itself does not match the “target require- 
ments” (or perhaps there were no requirements in the first place), but beyond 
exhibiting its proper world, it would also exploit the magmatic dimension 
of the world during the construction of this very world - due to the (poten- 
tial) exploitation of this magmatic dimension during “evolution”. Yet, what we 
have got now is a completely different concept of fabrication. There is almost 
no control anymore - neither of the “result”, nor of the schedule of the process 
itself. And this leads us to the last issue. 

Populating the world with beings-for-itself is just a special, though very 
prominent, case of the emergence of otherness — poiesis. Thus, the question 
of the fabrication of life entails the question of the fabrication of poiesis, and 
as we have just seen, this implies a change of the concept of fabrication. With 
fabrication we have actually two choices: we can lean on the ensidic, or we 
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adhere to the magmatic dimension of being. In the first case we start creat- 
ing ensidic constraints, determining “primitives”, production rules, and try 
to minimise alterity — producing forms by repetition and difference. We try 
to occupy the magmatic dimension of the world, so to speak, and substitute 
it by our own radical and social-historical imagination, eventually blowing 
out the poiesis of the world. We should be honest: as cowardly as we are (as 
a species), this world would end up in an eternal return of the ever same 
— boredom. 

The second choice we have would be to exploit the poiesis of the world. We 
would “listen” to the world — intensifying the emergence of otherness, enjoying 
the fecundity, and subversively reinforcing the overwhelming proliferation of 
different strata of being, disrupting continuity and thereby undermining the 
totalitarian pretense of the ensidic. 

Fabrication of poiesis, then, means keeping open the surging forth of phy- 
sis: alloiosis. It works out to be simply waiting for the right moment, the 
kairos — with Gelassenheit. 


If you want - an ethics of in-formation. 
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Juilian Rohrhuber 


In spring 2002, the scattered fragments of a meteorite entered the 
earth’s atmosphere and plunged to the ground close to the 19th century 
Neuschwanstein Castle. The impact did not cause any damage, but was 
noticed by local observers and its trace recorded by astronomical camera 
networks.! After a reconstruction of the trajectory from the photographic data 
and some systematic search in the field, two pieces were retrieved on German 
territory. Some time later, following a corrected model of the expected shape, 
a third and largest fragment was found by a German physicist across the 
nearby Austrian border. Since these meteorites (of the enstatite chondrite 
type) are well recorded and of relatively high value, their material presence 
immediately caused a conflict between potentially rightful owners, such as 
the mayor of the small town of Reutte (by proxy), and the finder. The court 
case turned out to be intractable — the most heterogeneous categories of law 
had to be taken into consideration; after all, the laws that govern the interac- 
tion between heavenly bodies do not regulate the accumulation of wealth. 

Seen from a distance, all that had happened was a minor extension to the 
planetary material. However, according to extant law, the situation could not 
be easily decided. Is a meteorite like snow (which ‘falls’ under the responsibil- 
ity and property of the landlord), is it like apples from neighbouring premises, 
or like flotsam (which is regulated by specific laws)? Is it to be considered 
a natural monument? A report by the Bavarian state lawyer Kristine Faust 
discussed these issues circumspectly; she clarified that a meteorite is not 
material fallen from neighbouring premises, and that the ground it has fallen 
on has not produced it either.” Only something that is lost, can be found, 
and as the state had not acquired the meteorite in the moment of impact, it 
was not lost property either: despite the fact that gravity may be enough to 
juridically bind a thing to its premises, the meteorite was still light enough to 
be easily removed without the application of “disproportional effort”. Yet with 
the first fragment, Faust came to the conclusion that the case was analogous 
to the discovery of hidden treasure, a solution that led to simply cutting the 
stone in two halves of equal weight, one for the state, one for the discoverers, 
who divided their half and sold the fragments. In the second case, the claim 
by the Austrian town was delivered a rejection, culminating in the statement 
“there is no earthly right to heavenly goods”, and ownership was granted to 
the finder.’ 


1 The fish-eye camera propeller of the European Fireball Network scans a complete 
night sky every night. The photographic observation was published in Spurny et al. 
2003, pp. 151-153. 


2 Faust 2003, pp. 28-31 


3 It remained unclear whether the then rightful owners had to pay income tax for 
their new possession. 
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Affiliations between things and living beings take the most diverse forms; 
this theme could hardly be more quotidian — everyday life is occupied with 
houses, tools, vehicles, but also with the material reality of weather and grav- 
ity, and trying to separate environment from inhabitant may at any moment 
turn out to become a knotty issue. Also, nothing stops us from extending the 
realm of things to languages, signs and symbols, which can forcefully turn 
out to condition an umwelt, just as they can assume the place of implements. 
So when it comes to the origin of actions and intentions, this context depend- 
ency makes it necessary to consider interactivity as fundamental to any 
investigation. There remains a certain dichotomy here though: while objects, 
structures, rules, or laws depend on a particular timelessness, movements, 
processes, interventions are almost exclusively temporal. 

As the observation of scientific and artistic practices shows, it would be 
a mistake to locate the origin of attributes like intentionality or initiative in 
the human mind alone. Social, material, structural circumstances force deci- 
sions, just as they are subject to modification and investigation. It is possible 
to avoid a foundational choice between a social constructivist and a real- 
ist view by a different account of objectivity. This objectivity is the result of 
interaction. Such “situated knowledges”, as Donna Haraway writes, “require 
that the object of knowledge be pictured as an actor and agent, not as a 
screen or a ground or a resource, never finally as slave to the master that 
closes off the dialectic in his unique agency and his authorship of ‘objective’ 
knowledge.”* In this context, it is significant that many theories seem to have 
shifted agency toward objects instead of looking for it in processes. In Alfred 
Gell’s anthropology of art, for instance, the main agent is the material art 
object.° For Donna Haraway, and also for Michel Callon and Bruno Latour it 
is the various hybrid, yet material coalescences that defy categorisation as 
either natural or social, animate or inanimate; they take the shape of collec- 
tives, ‘agencements’, which, despite their multiple forms, tend to crystallise 
in matter. Hans-Jörg Rheinberger proposes an intermediate form between 
concept and object, the epistemic thing.’ 

Other than simply resisting common preconceptions, there are also 
good reasons for applying the notion of autonomy to material objects rather 
than to processes merely because the latter appear closer to ‘being alive’. 
Firstly, objects imply specific actions and inherent necessities — they can be 


A 


4 Haraway 1988 
5 Gell 1998 


6 Haraway 1988; Latour 1993; referring to Donna Haraway, Callon writes, “These 
agencies, like Hobbes’ Leviathan, are made up of human bodies but also of prosthe- 
ses, tools, equipment, technical devices, algorithms, etc. The notion of a cyborg aptly 
describes these agencements.” (Callon 2005) 


7 Rheinberger 1997 
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described in terms of constraints and resistance, both physical and social. 
In a sense, things implicitly encode processes. A ball in a game is a clas- 
sic case. Secondly, an object occupies a place, may be attached to owners, 
can be passed on, and is thus able to transport action patterns between 
nodes in a network of relations. It may itself become a node, being equally 
the subject of, and subjected to, new formations. Not to mention that things 
can be traced, and sometimes collected. Nevertheless, according to agency 
theories, it would be wrong to treat objects (as well as subjects), as primary to 
their relationships with each other. Rather, it is the association between agent 
and patient that results in the resistance, the stubborn, ‘objective’ ignorance 
toward change that causes objects to exist, somewhat like the apparent stasis 
of an eddy or vortex in the flow of a stream. These linkages, which French 
sociology termed operation chains? are transactions of potential action.’° In 
this capacity, they are agents with social leverage. Because they result from 
these chains, objects are inherently political. 

The decision to put aside the essential polarity between the agency of 
persons and the agency of things allows us to treat the collective situation 
as existing logically prior to subject, action and object, and to render them 
conceptually indistinguishable. Deliberately creating a mode of observation 
with a blind spot for these distinctions causes new differences, subjects, and 
situations to appear. Areas of thought that tend to suffer when actions are 
merely considered as transmission from internal intention to external expres- 
sion can be better explored by not presupposing objects to which intentions 
and actions of subjects can be moored. Thus, issues of intentionality in art 
and the intricate relation between discovery and fabrication that constitutes 
objectivity in the sciences cease to disturb investigation. In what way, for 
instance, does an artwork participate in the possibility of its own forma- 
tion or condition its own becoming as it unfolds? How does a discovery turn 
out, after the fact, to constitute the very place it must have been part of 
already? In such issues, rather than a hindrance, paradox and undecidability 
turn out to be the driving force that opens previously inconceivable possibili- 
ties. “Experimental systems,” Rheinberger writes, “[...] allow researchers to 
arrive at unprecedented, surprising results. In this sense, such systems are 


8 For Michel Serres, the football is a good example for what he calls a ‘quasi-object’. 
His figure ground reversal illustrates how agency and patienthood may swap places: 
according to this perspective, in a football game, it is not the players who control the 
object. Conversely, is it the ball that is the subject of circulation between stations, and 
the players follow after it. A quasi-object is only an object insofar as its movement binds 
a collective (Serres 1987; Roßler 2008). 


9 The term operation chain (chaîne opératoire) was introduced by Leroi-Gourhan in 
1964. It has its origin in archeology, where it had been developed out of the research on 
the action patterns in the fabrication of Stone Age tools, traceable from raw material, 
completed artifacts, and their chippings on production sites (Leroi-Gourhan 1964). 


10 Schüttpelz 2008 


‘more real,’ if you will, than ordinary reality. The reality of epistemic things 
is their resistance, their resilience, their capacity, as ‘jokers’ of practice, to 
force us to abandon preconceptions and anticipations.”!! In the following, 
I will discuss agencies that force the formation of a series of conjectures, 
of open hypotheses. Especially I will do so with respect to a kind of incon- 
sistent hybrid between representation and unfolding, which may be called a 
‘model’. A detour into a specific praxis of computer programming, interactive 
programming, will provide evidence for a specific kind of interactivity typical 
for experimental systems; it can be traced back to an agency in the formal. 
Here, interactivity will turn out to imply a rather simple, temporal paradox; 
instead of looking for it in an immediate ‘presence’ of coupling, this observa- 
tion will help to show how the unfolding of an investigation implies interactiv- 
ity between its own history and future. 


Program, model, trap 


Taking operation chains as causes underlying the formation of objects 
does not imply that these objects explicitly represent actions. Art objects, for 
instance, may be cunningly prepared in a way to cause a certain impression, 
and to inhibit others; they may function to impress the audience, or a patron; 
tools, or other objects usually imply certain actions, but are not self-explana- 
tory; such things are a part, or a trace of an operation chain, but do not give 
access to the chain itself. Nonetheless, there are also many cases in which 
operativeness is combined with its description e.g. calenders, maps, plans, 
recipes, algorithms. A program is one example of a thing with such a double 
nature: it explicitly provides a plan together with a method to actualise it. The 
text of a program represents two processes at the same time: in the context of 
a given computer language, it activates a computational process, which may 
(or may not) produce results. Second, and this is what is supposed to qualify 
a ‘good’ program, it causes a human reader to understand, in one way or 
another, this process. A program can be regarded as ‘operative writing’'?, as 
an assemblage of a possible process and its description. 

As in any language though, one should not expect a transparent transla- 
tion from formula to meaning, or a complete reflectiveness between process 
and description. The hybrid assemblage of operation-representation is neces- 
sarily incoherent. An example of a process that produces a description may 
illustrate this; since we have to describe this process of description-making, 
we can think of the ‘simplest’ case as a description which describes nothing 


but its own making. Such programs whose output coincides precisely with 
nn 


11 Rheinberger 1997, p. 246 
12 Kramer 1993 
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their own source code (quines) are often fairly hard to understand; in order 
to write a program that is a description of how to type out its own text, one 
usually has to construct a maze of means and ends, of ‘use’ and ‘mention’, 
of quoting and unquoting of quotes. This may be a clue that there are good 
reasons for the fact that the description of an operation chain resists super- 
position with the chain it produces. Here, the semantics of such a program 
is its code in a literal form.'? Yet from a different perspective, it describes 
the process of producing this very code; in other words, it is because object- 
and meta-language interrelate that makes a quine difficult; in less reflective 
programs, where means and ends are more separate, this difficulty is not so 
obvious. 

In a formal language, the semantics of a sentence, its meaning, is called 
a model. Also, a model of a whole language is all that can be ‘expressed’ in 
it. The model as such is a purely mathematical concept that usually refers 
to an abstract domain; yet at the same time, algorithms are, in a sense, the 
mechanical equivalent of a part of mathematical praxis. So it is justified to 
ask what a given program-text really means from a formal point of view — 
what is its model? With respect to our train of thought, three possibilities 
are obvious. Does it express (1) its result, which is the effect and endpoint of 
its execution? Does it express (2) the process that leads to this effect? This 
is not obvious at all. In fact, looking at the details of specification, this will 
always remain a slightly ambiguous issue;'* what seems like a description of 
a result (a domain) may take on a more operative aspect (a process) in another 
situation; in other words, operation chain and its effect can never entirely be 
disentangled. And (3), we may have to take the program literally and see in it 
an inscription of the programmer’s thought rather than a direct description 
of either process or result.'® Usually, these three levels are arranged in an 


13 Note that this may be any kind of representation, which need not be ‘text’ formatted 
in ASCII code. A quine in a visual programming language, for instance, would have to 
compute its own visual code as an image, without re-using parts of this representation. 


14 This difference can be formalised in computer languages, but most languages do 
not do this. Even with a formalised semantics, semantics remains a matter of decision 
(For a thorough discussion on algorithmic equivalence, see Blass et al. 2008). Formal 
systems like pi-calculus explicitly encapsulate semantics into the system, which is 
passed around between agents. Computer semiotics, on the other hand, emphasises the 
process of meaning production as interactive coupling between cultural and algorithmic 
processes (Andersen 2003; Nake 2003). 


15 Alternatives (1) and (2) correspond to two different understandings of formal com- 
puter language semantics: operational semantics refers to the computational steps of its 
process, whereas for denotational semantics (this term is a bit misleading) the process 
does not matter, as it refers only to its eventual outcome. Note that this outcome may, in 
turn, be a process, such as an interactive application. In most discourses, the meaning 
of a program for the programmer (3) is not considered separately. Nevertheless, empha- 
sising the human reader and the communication of ideas, the concept of literate pro- 
gramming brought forth by Knuth (1992) and Iverson (1979) requires code to be taken 
more literally. Andersen (2003, p. 190) even takes formal semantics to be “the rules that 
we employ ourselves to read a piece of program,” and the compiler a “machine execut- 
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instrumental relation; code simply expresses its result, not unlike a pocket 
calculator. This is because when a computation is fast and leads to some sort 
of unchanging entity, such as a number or the data of an image, the process 
can be thought to implode in the blink of an eye. 

The situation is a little bit different as soon as the program is to describe 
a process that unfolds over time. In the above case we can say that a reck- 
oner’s operational agency (and labour), calculating by hand, is replaced, and 
hidden, in the rules of the formal system. This is not so obvious when the 
program’s result consists in processes, possible behaviours, actions. Here, it 
is easy to lose the distinction between the program as a process that leads 
to another process, and this latter process (which is a program too). The 
‘application’ replaces the ‘program’; possible interactions with the application 
are identified as the ‘behaviour of the program’, so that now the assemblage 
between plan and process is solidified in a thing, such as an interactive appli- 
cation, but also as an interactive installation, augmented environment, etc. 
In the first case it was the effectiveness of the algorithm that made it possible 
to neglect the ambiguity between process and product, in order to command 
a view on the relation between timeless formula and its immediate computa- 
tional result. Here, it is the interactive computation that, by representing the 
behaviour of a possible agent, replaces the semantic ambiguity. On the one 
hand, the fact that now computation happens necessarily over time, makes 
it more obvious that a program is an automaton, which works exactly in so 
far as it has been abandoned by its programmer. The model is left behind; 
instead of being the originator’s operational result (like a painting is the art- 
ists product), it embodies the originator’s formal ghost. On the other hand, 
this autonomy also introduces the above double meaning between processes, 
where the ‘making of’ interactive behaviour hides in the runtime behaviour 
itself. The model becomes objective because it is, to a degree, independent, it 
is an operational proof of its own unfolding. 

This is a reason why attention has been drawn to the notion of model- 
ling as an ideological notion, both from epistemology and computer semiot- 
ics. While semiotics tends to suggest a constructivist position instead, where 
programmers create a reality,’ and the model of a program is a metonymical 
and metaphorical structure, within a materialist epistemology, programming 
(or formalisation in general) is taken as an experimental process with its 
own structural constraints and mathematical domain: “A formal system is a 
mathematical machine, a system for mathematical production and is placed 
within the process of this production.”!” Objectivity is not a function of behav- 


MLL 
able Representation of this.” To regard programs as narratives, or as discursive media 
has become more widespread today. 


16 Noble et al. 2002 
17 Badiou 2007, p. 43 
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ioural resemblance of an artificial object with the natural object of enquiry, 
but it is to be found in the stubborn openness of formal systems. Precisely 
in so far as the model has been left behind, it is an agent, and its behaviour 
subject to discovery. 

There is a well-known cultural technique, which differs from numerical 
calculation yet represent a type of reckoning that may help to clarify this 
issue, or at least may show a way into further investigation. In his 1996 
article Vogel’s Net, Alfred Gell gives an analysis of cultural praxis that bears 
many aspects of his concept of agency of artefacts which he published two 
years later. For Gell, animal traps are a peculiar kind of thing capable of more 
than it seems at first glance. Essentially, a trap is a mechanical implement of 
the hunter’s ability to catch or to kill. Its reactivity is a model of the hunter’s 
awareness, its mechanism is a model of cognitive competence. “It is, in fact”, 
he writes, “an automaton or robot, whose design epitomizes the design of 
its maker. It is equipped with a rudimentary sensory transducer (the cord, 
sensitive to the animal’s touch). This afferent nervous system brings informa- 
tion to the automaton’s central processor (the trigger mechanism, a switch, 
the basis of all information-processing devices) which activates the efferent 
system [...]. This is not just a model of a person, like any doll, but a ‘working’ 
model of a person.” Similar to Latour’s example of the ‘sleeping policeman’, an 
object takes the place of a human in the enchainment of causes and effects. 
Yet, Gell notes that, at the same time, the trap is not only a model of a hunter. 
A trap is an altered environment — not so much as the hunter perceives it, but 
rather as a portrait of the animal’s perceptual Umwelt. In order to catch, it is 
a model of an observer being caught. Gell notes that “[...] if we look at traps, 
we are able to see that each is not only a model of its creator, a subsidiary self 
in the form of an automaton, but each is also a model of its victim. This model 
may actually reflect the outward form of the victim [... or] the trap may, more 
subtly and abstractly, represent parameters of the animal’s natural behav- 
iour, which are subverted in order to entrap it. Traps are lethal parodies of 
the animal’s umwelt.”!® 

As a superposition of two models of complementary observers, I think it is 
plausible that the agency/patienthood of a trap resembles that of an interac- 
tive program. Its mechanism is a concrete abstraction, a passe-partout of its 
parameters. Like a trap, a program is “a model as well as an implement.”'° 
Program and trap both encapsulate a hidden, objectified plan that unfolds 
into a scene only at the appropriate circumstances. They are situated mod- 
els of a silent, absent observer which is present in the observation of an 
observer. The conditionality and sequentiality of an algorithm come to a halt 
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18 Gell 1999, p. 200 
19 Gell 1999, p. 200 
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when it snaps shut. But what is actually caught by a program? And who is 
the hunter really? Is there a moment of closure at all? The analogy could be 
a trap itself. 

Apart from notable exceptions, the process of interaction, as it obtains 
between a running program and its environment, seems to have little of the 
sudden abduction by a mechanical implement. And when it is not meant 
to ensnare a potential person, but rather to give access to a new situation 
and unforeseen observations, it cannot be simply a portrait of either cogni- 
tive hunter or cognitive prey. Rather it appears to shift between an instru- 
mental aspect, where computation simulates physical processes or enables 
communication, and a more oblique situation, in which a given causality is 
disturbed, where it is not clear what actions find continuation and which 
percepts are consequences of the local logic. One aspect is easily forgotten 
though, when looking at real-time interaction in art, or also in scientific sim- 
ulations. Because, unlike in the early days of computing, human computer 
interaction today involves mostly the relation between algorithmic processes 
and users, the activity of constructing a program in the first place is taken as 
a preliminary means for creating interactivity. More precisely, if we follow the 
chains of causation in the loop between the various participants, we find that 
the algorithmic process is like a parallel world, only accessible through exper- 
imentation within the premises of this specific set-up. The mechanism’s con- 
structive preconditions stay hidden and become apparent only in the agency 
of its behaviour. Much of the critical effort within media art has been aiming 
toward bringing these conditions into discussion, making them accessible 
and contextualising them in the politics of things. This is a broad field, since 
the computational chain potentially pervades the situation just as much as 
cultural meaning passes through the networks of calculations. In order to 
reason about the conditions of interactivity, it is necessary to expose not only 
the model (be it process or result, or further chains of semiosis), but also the 
model formation within the interactive situation. 


Finding out 


Together with the concept of interactivity comes the notion of real-time. 
The fascination of self-regulation in the early cybernetic discourses is implic- 
itly connected with a coming to life of a dynamic continuum of becoming. 
Also, over most of the 20th century, digital systems were somehow always too 
slow — the resistance to interaction took place in an interval between input 
and output. A continuous labour to integrate computational systems into the 
environment, for instance in the form of scientific simulations in the work- 
flow of a laboratory or in the form of interactive music instruments, led to 
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the reduction of the delay in the loop to the point that it became sufficiently 
short to give the impression of a neutral presence of time. Yet, simultane- 
ously, measurement and display became fundamentals, which continuous or 
discrete interactive processes were to operate upon. More precisely, we can 
state that real-time interactivity is predicated upon the idea of the parameter. 
Pivoting on approximations of real numbers, real-time computation consists 
of a network of connected streams, whose immediacy is mediated through 
a parametrised mask of measured movement, sensor information; often, a 
graphical interface for such applications gives access to interaction points by 
means of images of sliders and wheels. More generally, a parameter-space is 
the implicit frame of reference in real-time interaction. 

Interactive programming has been taking the complementary approach: 
instead of writing an interactive program that exposes continuous param- 
eters at runtime, it exposes the activity of parametrisation itself, and more 
generally, the construction of programs at runtime. A starting point may be 
a very small formula - for instance a sound algorithm, which generates a 
process that, by converting it into an alternating current and playing it over 
a speaker, can be listened to. Instead of now thinking about what parameters 
need to be exposed to external change, and building an interactive applica- 
tion that can be used later, the formula that describes the process is rewritten 
directly. Changes of the program’s time-map figure as the medium of interac- 
tion. Therefore, in such a situation, it is not so much the parameter space 
that is subject to experimentation, but the program text itself. More precisely, 
as we shall see, it is the different semantic levels of a program, which become 
thematic again. 

In experimental mathematics, methods of interactive programming have 
been used to investigate the relation between a program text (a formal expres- 
sion) and its output, which may be a set of numbers, or other formal expres- 
sions; within application design, such methods allow iteratively improve 
computer applications. Usually, each version of the program text simply 
expresses one such relation — one description corresponds to one result. In 
the conversational approaches of the 1960s for instance, an incomplete pro- 
gram would ask the programmer questions until the result was found.” Now, 
as we saw before, this outcome may not have to be static. It also may be a 
process that exists only insofar as it unfolds and changes over time. In algo- 
rithmic sound synthesis, for instance, the program text describes something 
irreducibly situated in time; of course, one may record a sound wave and later 
play it back, jump around in it, or play it backward. Nevertheless, this does 
not touch the relation between an algorithm and its unfolding. Only after the 
fact, is everything data. 


ZA 


20 Matthews 1968; this procedure survives in today’s terminal application, where pro- 


gram and programmer interact by turn-taking in the form of a dialogue. 


Taking a closer look at the structure of interactive programming, it becomes 
apparent that experimenting with inherently temporal results requires the 
relation between description and model to change significantly. Generally 
speaking, this is because the program as a description cannot consistently 
represent the changes to this description themselves. It turns out that should 
a program be rewritable at runtime, we are confronted with a paradoxical 
situation. If we start the whole computational process from the beginning 
each time something is changed, then the formula really can count as a valid 
plan of its unfolding. But each new onset means that the change itself is no 
longer situated in the moment it actually happens relative to the ongoing 
process. The world ends and is recreated. Another procedure is to divide the 
program into concurrent parts, each of which can be changed individually, 
but which may interact over time. Then, when a part is changed, it is the local 
unfolding of this new part that affects the rest of the system. This sounds like 
a good solution, because then, changes to the code happen in the context of 
a continuous behaviour of the system. But now we are confronted with the 
fact that the description does not reflect the behaviour as a whole anymore, 
since different parts of the system must be understood relative to their dif- 
ferent points of departure. Moreover, looking at the program in its entirety, 
an assemblage of parts, it is not always clear whether they are a description 
of how to put together other parts or whether they describe the behaviour of 
such a part.?! As soon as one tries to integrate the rules into the interaction, 
their divergent interpretations (expression, process, or result) come into play. 
Of course it is possible, as a next step, to formalise the structural transition 
between different descriptions and represent those as the program. We then 
still have to decide which part of the text belongs to what part of the struc- 
ture. Even more, the structure of these changes itself is again subject to the 
same problem if it is meant to be part of the interactive situation. So while 
some kind of segmentation is necessary, it is not generally decidable what 
belongs together and what is separate. 

This symptom, which arises within interactive programming, allows us 
to consider the implications of unfolding on a more general level. As we have 
seen, the instrumental relationship between description and process, when it 
is supposed to allow an interaction in real-time, comes with a specific exclu- 
sion of the process of constructing the same system. If an immediate coupling 


21 The moment of substitution of a part by a different part dissipates the general 
ambiguity of semantics of a whole program within its own parts. See e.g. Abelson/ 
Sussman 1996, ch. 1.1.5: “Despite the simplicity of the substitution idea, it turns out to 
be surprisingly complicated to give a rigorous mathematical definition of the substitu- 
tion process. The problem arises from the possibility of confusion between the names 
used for the formal parameters of a procedure and the (possibly identical) names used 
in the expressions to which the procedure may be applied. Indeed, there is a long his- 
tory of erroneous definitions of substitution in the literature of logic and programming 
semantics.” 
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of a system is desired, the laws that regulate this coupling cannot themselves 
be subject to interaction. In order to include these laws, not only has the 
idea of real-time to be relativised, but even more, the meaning of the system, 
the concept of a model becomes a matter in question. As long as we know 
exactly in advance what the meaning of a program is supposed to be (as long 
as we have a good specification, e.g. of its behaviour), however difficult, there 
is a possibility to line up the chain between description, computation and 
result, and construct an appropriate formalism. However, trying to find anew 
model requires a re-ordering of the whole situation, because interactivity with 
rules (and not with their parameters) entails a multiple split in the temporal 
domain; where time ceases to be a ‘domain’ that could be called ‘real-time’. 
The ambiguity between things and their operative formation reappears here 
in form of conflicting levels of meaning, which can only partly be disentan- 
gled by situated decisions.” Interactivity - maybe as opposed to interaction 
— turns out to be an open temporal antagonism, a differend”* on the verge of 
multiple temporal levels and multiple possible models. In a sense, this makes 
interactive programming thinkable as a miniature version of an experimen- 
tal process on the border between formal and empirical methods, iterating 
between the different mechanisms of explanation, conjecture and failure. It 
exemplifies that when no cogent specification is given, and the subject of 
investigation is ambiguous, we cannot line up formalisation with interactivity 
in such a way that the former is only the necessary sacrifice to the technical, 
whereas real-time behaviour is authentic becoming. A situation in which it 
should be possible to find out something, the delimitation between thought 
and act has to be made amenable to reassembly. In other words, since it is 
not obvious to what degree the outcome is a construction of a new structure 
or a discovery within the current one, an investigation concerns the relation 
between free decision and strict deduction, or, from a different perspective, of 
the rational and the social.* 


22 Inspired by Lyotard’s article ‘Time Today (Lyotard 1991), in our paper ‘Algorithms 
Today. Some Notes on Just-In-Time Programming’, we have discussed this multiplic- 
ity of history in the context of a concrete system for interactive sound programming 
(Rohrhuber et al. 2005). 


23 To give an example from empirical science: If we have an idea what parameters of 
some physical process may be relevant for a law, for some invariance, we can measure 
them (if we are lucky), and then test our equations against this data. Should either the 
experiment or the theoretical framework inspire us to see some other possible parameter, 
the measurement has to be done again. This is how the interaction between material cul- 
ture of science and its concept formation is usually explained. (See e.g. Pickering 1995; 
Rheinberger 1997). In other words, prediction depends on the past, yet at the same time, 
the significance of past facts depends on their future effects. This causes interactivity 
to be necessarily situated in incommensurable orders, or incompatible law systems. In 
Lyotard’s terminology, it can thus be considered a differend. 


24 Longino 2001; already the early discussions on interactive programming show con- 
sequences that this has for the place automata should have in a process of reason- 
ing. In the introduction to the proceedings of the conference Interactive Systems for 
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The temporal aspect of this problem can be found in the relation between 
rules and their unfolding. Formally, it can be described as the relation 
between a structure and a model; more precisely, between some kind of for- 
mal deduction laws together with basic assumptions, the axioms, on the one 
side, and on the other, some mathematical domain, e.g. sets. Evidence sug- 
gests that formalisation is not only the necessary precondition for an inter- 
active, experimental, empirical investigation; rather it is already part of this 
investigation. The construction of a logic requires mathematical assumptions. 
Or, for instance, within algorithmic composition, programming is not just a 
technique for building synthesisers, but is part of the compositional process. 
Similarly, in scientific operations, the schematism is not a precondition for 
empirical confirmation, but both are part of a new form. We have seen that 
this hybrid, this mutual implication between formal apparatus and empirical 
praxis is not simply a fusion. Rather, the structure of an investigation must 
be regarded as a paradoxical interaction between what is possible and what 
becomes possible by doing the possible. This “dialectic of formalization””° 
unfolds in a mutual determination of what is given and what is found out: 
“every creation of thought is in reality a creation of a new formalization and at 
the same time this new formalization establishes a relation or takes partinan 
interaction with the particularity of what we are trying to express.”° 

The process of finding out something operates in-between a discovery of 
something that previously existed, forming the conditions of research, and, at 
the same time, the construction of a new situation that did not exist before, 
but reconditions what can be constructed.” The model is a linkage between a 
new possibility and a situated context in which it becomes unavoidable. 


Experimental Applied Mathematics (Klerer/Reinfelds 1968), Klerer quotes Burton Fried, 
who had written one year earlier that the “utopian notion of a computer, which accepts 
the statements of a problem and automatically finds a way of solving it is clearly chimeri- 
cal, save for those ‘problems’ whose structure has been thoroughly understood and for 
which methods of solutions are well known” (Karplus 1967, p. 169). Again, Klerer con- 
fesses that his own motivation, in contrast to the majority of the contemporary academic 
community, is “based on just such a utopian basis.” He emphasises that, while the term 
interactive is difficult to define, “[...] we would expect more than in the old process of 
inputting a well-formulated set of directions with the machine performing in its capacity 
as an idiot servant.” (Klerer/Reinfelds 1968, p. 9). I think that quite conversely, Fried’s 
comment alludes to the basic incompleteness of a majority of formal systems, showing 
that exactly because we may extend a given system by a term that is not derivable from 
within it, interactive programming is interesting. One cannot decide in advance what 
part of the system will turn out to be involved in such a change; this suggests interactive 
programming (of whatever kind) as an interesting conceptual alternative to interaction 
with a program that has its interaction point already defined in advance. So problem 
solving is indeed “chimerical”, since it involves ambiguous agencies, which explains the 
need to include the programming activity itself into the program. 


25 Badiou 2007, pp. 90-92 
26 Badiou 2007, pp. 90-91 


27 Inthis, the dialectics of formalisation are equivalent with Badiou’s later concept of a 
truth procedure. Following Cohen’s mathematical technique of forcing, Badiou is able to 
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Traps revisited 


Computation follows a strict protocol laid down in its description, yet at 
the same time gives rise to completely unexpected things. This is why an 
algorithm may, under some circumstances, occupy an intermediate position 
between a law as discovery (when we find it, we suppose that it has always 
existed) and the law as a constructed artefact (which is an intervention into 
what exists).”® It becomes a hybrid between the ghost of the programmer, 
stood in for by the automaton and the autonomy of a new situation. 

Let’s return to the question in how far interactivity of this kind is struc- 
tured in analogy to a trap. What is caught in an algorithm? Who is it an agent 
for? I think the interesting central thought that Gell started out with - namely 
that in some way, the trap is a model of both hunter and hunted - is useful 
for clarifying the situation of a ‘dialectics of formalisation’ that implies some 
paradoxes of interactivity. In his investigation of agency, Gell is able to show 
that certain situations and artefacts cause the observer to enter into a proc- 
ess of reasoning.” Their questionable mode of fabrication, the unclear origin, 
maybe we can say their artificial and alien character, causes them to force 
an attribution of agency. “Is this spot here on purpose or did it just happen 
unintentionally?” or “Is this strange sound we just heard part of the composi- 
tion, or is it a mistake of the performer?” Agency in this sense is essentially an 
open question provoked by a disturbance of conventional inference, a ques- 
tion which can only be answered by hypothetical reasoning, or, as Gell puts 
it, abduction of possible originations. This search for the inner logic of a situ- 
ation enmeshes the participants in possible alternative worlds of causal, and 
thus, temporal connections. 

In a peculiar dialectics, agency is what an observer infers of a phenom- 
enon’s origin, and simultaneously it is the power to induce this reasoning. 
Like the trap, the artwork is a disturbance of the causal milieu in which it 
is situated,” a disturbance that opens an explanatory gap, a cognitive dis- 
sonance. We can say it is a different model of cause and effect within a given 
reference frame, a model that forces one to hypothesise about possible expla- 
nations (this abduction is the derivation of a law from a model). In this way, 
certain artefacts are able to induce interactivity — interactivity as a process 


show ontologically under what conditions a new formalisation is possible in a given situ- 
ation (Badiou 2007b). For an investigation of a relation between set theory and agency 
theory in this light, see Rohrhuber 2008. 


28 The issue unfolds in a contradiction between place, finder (an investigation), and 
ownership: does the found object belong to the place or the finder? Does the place belong 
as much to the landlord as the object belongs to the finder? Does observing an event and 
investigating its traces set a rupture that contradicts the continuity of territory? 


29 Gell 1998 
30 Gell 1998, p. 20 
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of reasoning, of situated thought. That human beings may make such infer- 
ences is without question. But it is more interesting to ask what could be the 
conditions under which a new formalisation, a new causation may appear at 
all. As we have seen, in such a rupture, social and natural causes become 
mutually exchangeable, just as much as the difference between construction 
and discovery of reality have to be negotiated anew. So if an artefact, or more 
generally, a situation as a whole may be the cause of such a shift, how does 
it have to be structured? What is a model for finding models? 

As a conceptual starting point, traps have turned out to be interesting 
as an epistemic model; they allowed us to consider something like an objec- 
tified anticipation together with an objectified ignorance.*! At first, this was 
thought as two parts: an open conjecture and an automaton, a ‘materialised 
theory’ on the one side; and the unfolding scene of captivation on the other, 
where the enclosure must have already been entered before it snaps shut. 
It is in evidence that these, in turn, imply two concepts of time; in the first, 
the absentation of the hunter causes the prey’s possible presence. In the 
second, the prey is already caught before it realises this fact. Like in a weir, 
for a fish there is no point where the difference can be found between inside 
and outside. In other words, in order to discover something, and not invent 
it, this entity must have some autonomy; however artificial the situation, it 
must show itself. Yet it must show itself in the situation that is given already. 
However if it is not certain what is to be caught, this separation becomes 
unstable, and as a consequence, the trap begins to resemble an experimental 
system. Here, the model breaks; hunter and prey become indistinguishable. 
Anticipation becomes a conjecture about a possible new situation in which 
we are entangled already. 

Thus, the trap remains a model for the possibility of finding out (some- 
thing). It suggests that it is the dynamics of laws that abstract from imme- 
diacy and allow experiment: the inherent temporal logic here is its formal 
indifference to time; just as abstraction allows statements not to differentiate 
between certain things, the abstract also shows indifference to the moment 
at which events occur. As we have seen, this is what made a program imply 
both possible actions and their formalisation, both operation chain and plan. 
But instead of giving rise to interactivity in the sense of a presence of unifi- 
cation, it has turned out that abstraction - if it is, despite all contradiction, 
included in the situation — leads to a resilient and antagonistic assemblage. 
31 For Blumenberg, quite in accordance with the view Gell proposes, traps are inti- 
mately related to the peculiar temporality and agency in concept formation. He considers 
preemption in its dialectics between absence and presence: “The trap acts in place of the 
hunter in the moment of his absence, but in the prey’s presence. These conditions are 
revealed to be the reverse in the trap’s production. It is the reified expectation. Insofar, 


the trap is the first triumph of the concept [Begriff].” (Blumenberg 2007, p. 14, my 
translation). 
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Suspending direct access, a formalised situation may itself impel the process 
of a different form. The abstraction from the trap allows us to maintain that 
hypothetical situations do exist: they are not just a product of an observer, 
opposed to a non-hypothetical world. There are cases where abduction is to 
be found within the situation — finding out such cases is itself a matter of 
formal experiment (it must then be possible to find something that was not 


even hidden). 
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S 1. Introduction 


S 
Pa While in the past the predominant goal of software engineering was the 
> Pr efficient development of robust, reliable and easy to use systems, the state 
S Pa of affairs seems to change radically at the moment. On the basis of gained 
E ¿$ experience and acquired adulthood, a significant number of users ask for 
S Pa empowerment. They not only want to be asked afterwards what they like 
ES or dislike about an application, but actively participate and get a say at all s 


S s stages of development, right up to the questions of profit-sharing. Therefore, ¢ 
< the recently checked out software design methodologies primarily try to 


s S 

l a s come up with strategies for participation on all levels of the software process. ra 

Ba Pa Packaging and shelving of software seems to be a discontinued model, the , 
f S new software development processes try not to end with the first deployment PA 

P of a system but stay open to the requirements of an ever-changing context Â 

à S 4 and the continuously upcoming needs of users. To achieve such systems we S 

E f Pa need open software architectures on the one hand, and the active involve- 

5 ge ment and the energy of participating people on the other. 

E i Â The development of the Unortkataster, which will be discussed below, is 

“é S driven by the intention to create an online tool in collaboration with users that s 

S Pai facilitates controversial thinking about critical places (Unorte) in an urban 

s d $ environment (the area of the City of Cologne). An “Unort” is a marked place $ 
& ae or area on a map, added by text-descriptions and other illustrating media a 

va S by one author or a group of authors. The term “Unort” refers to any urban Ea 
S yo place or situation, which is criticised for a special lack of quality - based on E 

a individual settings. The tool is intended to moderate discussions about these S 
Fa Fa S kinds of places and organise them by temporal and spatial aspects. In the fol- < 
r x lowing we analyse what potential the approach of ‘technology probes’ delivers 

E for participative software development in the context of big cities. This entails 
S Pr the question, which possible map-based, community applications may be S 
p E applicable in the interleaved domain of physically localised public space and ra £ 
i Pa a globally networked public sphere? v4 
og 4 In the following, we also try to illuminate another quite important — nor- oo 
r S s mally underestimated - aspect regarding participatory software systems. Once a 
oe developed and implemented, software acts independently from their creators. ` 

fe It becomes an active player in the social communication game. Algorithmic 

I Pa _ processes are self-operating entities that actively change the domain in which s 

S 4 Á they perform. This viewpoint does not focus on the cultural memory and/or vA 
E its well-investigated, modern realisation as digital archive. Rather it is about * 

EZ sg the underlying algorithms that select, copy, transform, compare, visualise, P 
Ef and permanently reorganise the digital information and its connected com- 2 
ri : vA munication processes. Thus, due to their invisibility and despite their omni- va 
Ea presence, algorithms are probably the most underrated artefacts of our days. “ 

s Ja va & d $ a 4 Fa Ea Ae 4 = S A 
ES Pa E G À s G é 4 rs 5 S SE a S 1 9 3 A 
DR a S a 5 S eS FF EEE E S 
oe fe & gE PELEP LE gf ¢ ge 
a * i s S 4 E S EEE ra oS S é å * 
$ P S se we = S 4 ER À fe S g E S p i À p as Pa 
E E S SEP ESS Sb S Fa E ER 
S Pi S g S 3 3 S g > se a $ # 


It seems to be time to ask a basic question: How do software algorithms 
actively shape society, its knowledge, and communication patterns? 


2. Societies and their technologies 


In his much-cited book Myth of the Machine, Lewis Mumford convinc- 
ingly demonstrated that human culture is not - as often stressed — mainly 
a result of man’s ability to build and command tools, but that tools could 
only develop so far because of a significant series of inventions in ritual, 
language, and social organisation.’ In fact, the first complex machines in 
history were not mechanical entities but composed of living humans, each 
assigned to his special office, role, and task, that led to - even under contem- 
porary criteria — tremendous work performances (e.g. the pyramids, palaces, 
town walls, etc.). Because the components of the machine were working in 
separated locations, although their interplay implemented an overall func- 
tionality, Mumford called them invisible machines. The labour machines were 
merged with the destructive military machine and the controlling adminis- 
trative machine to the megamachine of the early totalitarian states. Within 
this system we encounter the very first, really powerful machine operator in 
history. In this image, the whole society is the machine and the god-king has 
complete control over the machinery. “But only Kings, aided by the discipline 
of astronomical science and supported by the sanctions of religion, had the 
capability of assembling and directing the megamachine.” So the myth of the 
machine and the cult of divine kingship rose together, as Mumford puts it. 

The invisible machine of the ancient divine kingship was only manage- 
able because of its strict hierarchical organisation. And hierarchy is still the 
most powerful and important principle we have at our disposal to realise reli- 
able and controllable complex machinery. Thus it is not surprising that we 
also find this principle within the most complex (and maybe most powerful) 
machinery of our times, the computer. From an engineering point of view, the 
computer is only manageable because of its consequent hierarchical organi- 
sation. But what has been adopted from past totalitarian states and still 
is valid on a technological level has completely changed on the social level 
where the computer becomes effective. The organisation of society, the whole 
socio-economic and political structure as well as the accompanying power 
relations, is today based on the (more or less) free decision of participating 
humans and therefore completely different to the times of divine kingship. 
But also freedom can be considered as part of a computable function, as it 
was attempted by Stafford Beer in his failed experiment of the seventies to 
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1 Mumford 1966 
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transform the whole Chilean political economy into a self-organising, cyber- 
netic machine.” The substitution of hierarchy by self-organisation was an 
important paradigm shift in the modelling of societies. 

According to the sociologist Manuel Castells, we now live in the Network 
Society, which also only marks a transient state in an ongoing process of a 
renewed radical change of the power relations. “Cultural battles are the power 
battles of the Information Age. They are primarily fought in and by the media, 
but the media are not the power-holders. Power as the capacity to impose 
behaviour, lies in the networks of information exchange and symbol manipu- 
lation, which relate to social actors, institutions, and cultural movements, 
through icons, spokespersons, and intellectual amplifiers.”” This means 
that in the upcoming society, power does not simply disappear but becomes 
inscribed in the cultural codes through which people and institutions repre- 
sent their interests and arrive at decisions. The most important cultural code 
in which power will be inscribed is probably software code. 

What we should learn from Mumford, Castells, and others and what 
is still not sufficiently considered within the software engineering faction, 
is the insight that major technological progress always goes hand in hand 
with social dynamics. We can no longer restrict the scope of our software 
development projects to the characteristics and technicalities of the intended 
applications, but have to consider the whole social universe in which the 
application will function. The necessity to consult social aspects and the 
needs, life-styles and desires of people during software design processes is 
also supported by another novelty of the software design problem. This has 
to do with missing experience in our dealing with computational objects. 
“When designing a classic object such as a chair, there are long traditions 
embedded in the practices of designing and using chairs that are not easily 
escaped. When designing a computational thing, however, not only the object 
but frequently the entire object category will be new to us. The effect is that 
our understanding of the objects we set about designing is extremely limited. 
Lacking such a fundamental understanding of the object, studies of possible 
situations of use, the needs and desires of potential users, and methods from 
other domains of practice have become tools for navigating an unfamiliar 
design space.” Thus engineers who participate in the design and develop- 
ment of the new technologies will constantly — consciously or unconsciously 
— have to construct hypotheses and expectations about their users and the 
social context in which the technology will be used. Involuntarily, software 
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engineers thereby become sociologists, or engineer-sociologists as Michael 
Callon calls them. 

All these arguments lead to the conclusion, that, if we want do develop 
participatory media where people take over substantial parts of the responsi- 
bilities for the vividness and progress of the whole application, we have to find 
answers to some fundamental questions. Some of these questions are: What 
are the important contemporary rituals, the rules of communication, and 
the organisational forms of the involved communities and institutions? How 
does the social fabric of modern communities work and what is the role of 
software in this network? How can software show its own social conditionality 
and the wide influence of the subjective perspectives of the participants? How 
does software become effective in reality and change the communication and 
knowledge of the social environment in which it is operating (e.g. the Google 
page rank algorithm)? How can software remain open to the variations of the 
context in which it is performing and to the changing and proceeding needs 
of the users? 


3. Urban investigations 
and participatory processes in urban design 


In order to find answers to some of these questions the city appears as the 
condensed laboratory of society where changes become evident and future 
developments can be anticipated. The city still represents the most success- 
ful form of human co-existence and has resisted many announcements of 
its crisis. Contrary to former assumptions, communication technology can 
be regarded as a stabilising factor for the physical structure of the city. “The 
physical synergies between telecommunications and physical networks mean 
that both tend to concentrate in the same city centres and the same corridors 
between them.”® 

Condensation and the related diversification is an old motive for the appeal 
of the city and has a crucial impact on the social structure. Cities do not pro- 
vide a homogenous environment where people share the same attitudes at the 
same location. Instead, people of different ethnic and cultural backgrounds 
coexist and are able to live in close proximity to each other without being con- 
fronted directly. The guarantee to maintain differences in spite of the lack of 
geographic distance seems to have become a particular phenomenon for the 
modern city society. Moreover, the city allows the individual or community 
to erode the mechanisms of communicative constraints that are character- 
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istic for rural societies.” The city in this sense should be regarded as space 
where people might start talking to each other but might also refrain from 
doing so, as Dirk Baecker puts it.* The containment of the citizen contradicts 
efforts to change city culture by communication systems in order to enforce 
understanding among citizens and create a feeling of togetherness. Normative 
theories of urbanity therefore polarise communications into a “for or against” 
and try to enforce a consensus for one of the alternatives.” However the city 
as a physical structure is characterised by the production of conflicts that 
touch questions which go beyond the individual sphere. Wikipedia describes 
a community as “a social group of organisms sharing an environment, nor- 
mally with shared interests.” While in online communities shared interests 
are indeed the dominant connector, in cities the shared physical location 
becomes the other dominant condition for community building. Civic com- 
munities in this aspect are not primarily connected by shared similarities but 
by shared problems. 

However, the process of community building in real space evolves much 
differently from how it happens in an online environment. Online communi- 
ties have begun to conquer the former, exclusive territories of the city: portals 
like MySpace or Facebook have started to compete with the street or the 
mall as the preferred place to “see, be seen, and connect”. However, com- 
munity building in the city may emerge deliberately as much as accidentally. 
Communities therefore evolve because people share the same housing areas, 
the same thoroughfares, or the same recreation spaces. In addition the rela- 
tions of communities are changing over time or may even be time-based: peo- 
ple joining a traffic jam, people sharing the experience of a noisy street, peo- 
ple being involved in the effects of a natural catastrophe etc. Environmental 
aspects become connecting points as well as infrastructural connexions. 
Participative processes allow a better understanding of how the character 
of those communities is changed by the confrontation of physically localised 
space with a globally networked space. 

An early forerunner of participatory city-design, Kevin Lynch, tried to 
involve people into city planning processes. For that purpose he used simple 
but effective instruments: He asked citizens for sketched mental maps and 
did interviews at the same time to better understand the relation between the 
participants maps and the explanations. By using these very simple and com- 
municative methods he tried to understand how the individual image of an 
environment is constructed. He aimed at a better understanding for the “col- 
lective image”, which should finally lead to better city-design methodologies. 
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In his book The Image Of The City he summarises the requirements for an 
improved image: “The image should preferably be open-ended, adaptable to 
change, allowing the individual to continue to investigate and organize reality. 
There should be blank spaces where he can extend the drawing for himself. 
Finally, it should in some measure be communicable to other individuals.”!° 

Lynch’s concepts have remained a challenge because the act of design- 
ing a city from the very beginning is a rare possibility. Architecture proceeds 
in a dictatorial!! way because it forces the city into a structure of temporal 
persistence. The built structure is static and resists being adjusted according 


Fig. 1. Kevin Lynch’s analysis of »problems of the boston image« resulting in 
a mental map (Kevin Lynch (1960): The Image of the City, Cambridge: MIT Press, p. 
24, Fig. 8) 
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to changes in society. New architectural and city planning concepts do not 
lead to flexible structures that allow people to modify environments related to 
changing life conceptions. 

But the requirements Lynch verbalised for the rebuilding of cities can be 
read as a rationale of how networked communication systems are used by 
citizens today and which change the perspective of how openness of the city- 
image might be established by building into software!?. Communication net- 
works are working on top of built city structures and allow interaction inde- 
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pendently of physical barriers. Cell phones give the most obvious example 
of how this is influencing the behaviour of the citizens: making phone calls 
without depending on temporal and spatial constraints changes the rela- 
tions between participants as well as the relations between public and private 
territories. Shopping by the Internet allows not having to go to real shops 
and leads to new business models. Positioning systems suggest alternative 
routes for road users and changes traffic. Openness under these conditions 
means that the built structure loses its character as a communicative bar- 
rier: People’s activities and communication are liberated from the conditions 
of real space by the flexibility of the network. But to get an impression of how 
these changes influence today’s citizens’ image of the city, a proper analysis of 
impressions and opinions that are related to it would still be necessary. 

The mental maps that Kevin Lynch used to collect input from citizens 
during his investigations are becoming effective again in the context of city- 
related, online community tools. Transformed to software, they can become a 
powerful tool for the involvement of citizens in the process of finding common 
requirements for future developments. 


4. Software in social context 


This section will roughly describe our approach to software design that 
we pursue in the development of the Unortkataster. As a process of software 
development, this approach must partially rely on the history of designing 
both software systems and their interfaces. From this history we borrow ideas 
and methods and apply them within the context of participatory media and 
therewith the perspective of the engineer-sociologist. 

Firstly, to build such a participatory medium we need a strong engineer 
approach that not only leads to robust, reliable, and easy to use software 
architectures, but also allows for an easy adaptation to ever changing social 
contexts and the accompanying drift of user needs and expectations. Here 
we have to fight a serious paradox. We know that the context of the applica- 
tion will change, but we do not know how it will change. Nevertheless, our 
software has to be prepared for it. To encounter this problem we try to trans- 
form and apply laws from Software Evolution. Secondly, we need transpar- 
ent strategies of communication, moderation, documentation, and decision, 
which allow user participation during all stages of development. This entails 
the problem that the processes organised by the system must be understood 
in their changing subjective and social dimension and context. Thirdly, this 
involvement has to start before the first ideas are brought down to paper or 
any decisions about the application are made. This usually generates another 
paradox: Potential users (this term is already difficult) in most cases have no 
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clue how novel and innovative participatory media could look like, but that 
is exactly what we ask them for. Designers and users likewise need inspira- 
tion that is driven by technology and its possibilities as well as the needs and 
practices of its situated users. 

We may roughly cluster these problems into three categories: 1) engineer- 
ing, 2) participation, and 3) inspiration. The following brief history of interface 
design is oriented along these three terms. 


4.1 A brief history of interface design 


For software to become effective in reality in an immediate sense, an 
important technological change had to occur: the introduction of the inter- 
rupt. The interrupt constitutes a computer’s ability to react to new input dur- 
ing program execution. Although it has accompanied computing machinery 
since the late 1940s, computer science still struggles with the problems it 
poses for the theory of computation, as the theory of Turing Machines does 
not generally account for such interventions.'? For the application of comput- 
ing machinery, the interrupt established for the first time a real-time cou- 
pling between computing machinery and the world. Another change taking 
place around the same period similarly shaped what interactive computing 
remains until today: with the introduction of digital computing unlike analog 
(computing and other) machines, the relation of digital data to its represen- 
tations became arbitrary, or in terms of semiotics: not longer indexical but 
symbolic.'* Since then, the symbols through which users may perceive and 
act upon an interruptible computer have a two-sided nature: they are sign 
and signal - open to interpretation but at the same time causally effective in 
a machine." 

Following these changes, operators operating computing machinery 
became users wielding tools. With that, a science of users and interfaces 
emerged that tried to solve the problems raised by the changes, a science that 
from the 1980s on was called human-computer interaction or HCI. 


4.1.1 Early HCI: Engineer and test 


As the construction of computing machinery was an engineering activ- 
ity, HCI was strongly influenced by engineering disciplines. But since the 
interrupt allowed the incorporation of the user into computations, it was 
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also, from its very beginning, a science of people. It hence co-developed with 
another young, scientific discipline trying to understand the human mind: 
cognitive science (and artificial intelligence). And while cognitive science was 
largely influenced by work on logic and hierarchical problem solving, human 
use of interfaces was seen as a process of goal directed problem solving, too 
-a process that could be modelled and calculated in order to build ideal inter- 
faces for the “human processor”.!® 

For the engineer, who since the days of divine kingship has been work- 
ing with the principle of hierarchy, the process of creating software could 
likewise be organised as a process of hierarchical problem solving, starting 
from requirements defined in the beginning, and ending with a product that 
fulfils these requirements. The model that, until today, is associated with 
this process for its hierarchical, top-down structure was named the “water- 
fall model”!”. Since the requirements of an interface can only be fulfilled in 
conjunction with a user, soon usability tests with users were introduced - a 
method later named usability engineering. 

Because of the close connection of HCI, cognitive science and artificial 
intelligence, it is not surprising that the first influential criticisms of the way 
user and interface were conceptualised stem from books that were written as 
contributions to the artificial intelligence discourse. The book Understanding 
Computers and Cognition'® attacked the assumptions much work in artifi- 
cial intelligence rested upon by confronting this “rationalistic tradition” with 
Heideggerian hermeneutics and speech act theory. Thus the authors stressed 
the problems implied by context dependency and situatedness. The subtitle 
of the book already suggested the role it should later play, outside the artifi- 
cial intelligence discourse, when it was adopted by HCI researchers: “A New 
Foundation for Design”. 

Similarly Lucy Suchman in her book Plans and Situated Action also used 
Heidegger’s terminology to show that human actions and goal-directed prob- 
lem solving rarely coincide.!° Her theoretical considerations were backed up 
by a detailed empirical study of the interaction of users and an intelligent 
machine. This analysis was based on the methods of ethno-methodology. 
Here, too, what was introduced to the artificial intelligence field as a means 
of criticism was soon taken up by the HCI community and used as a design 
method. 
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4.1.2 Understanding situated use: 
Participatory design 


The critique formulated by Winograd and Flores or Suchman with its 
focus on understanding the situated human was paralleled by a development 
in Scandinavia: Here during the 1970s the participatory design tradition was 
born. 

Participatory design emerged from a special historical condition: during 
the 1970s, 90% of the Scandinavian workforce were organised in unions. 
Consequently, the unions were granted influence on large parts of what deter- 
mined their members’ working conditions. In Norway, a co-determination 
agreement was signed allowing worker participation in the development of 
new workplace technology.” Central to the early projects on co-determina- 
tion was the Marxist notion of ongoing conflict between capital and labour, 
as well as approaches that tried to understand computer systems as socio- 
technical systems.” Against the observation that “democracy stops at the 
factory gates”? the vision of workplace or industry democracy was devel- 
oped.” The power relations within the factories were to be rebalanced and 
this had to include incorporating the workforce into the overall design of their 
workplace. 

In a series of projects in Norway, Sweden and Denmark, computer scien- 
tists and their prospective users developed new technology for several areas 
of industry. These projects first of all led to new methods of how to enable 
users to participate in the design of new technology as it soon became clear 
that just facilitating conversation between designers and prospective users 
in order to establish requirements was not enough.?* Tools were needed that 
would bridge the gaps between designers and users, allowing each to access 
the tacit and contextual knowledge of the other. Therefore, methods such as 
role-playing with mockups — non-functional, low-fidelity prototypes made, for 
instance, of cardboard — became an important design method.” Prototypes 
may here be seen as artefacts that provide a common language for designers 
and users, enabling and structuring discourse across cultural boundaries. 

This “Scandinavian challenge” yielded “a US response” that unlike 
the Scandinavian tradition was not fueled by Marxist ideas but by market 
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demands, where “the result of a good design is a satisfied customer.””” The 
methods developed within participatory design projects here proved to sat- 
isfy customers (or management representatives) just like workers’ unions in 
Scandinavia. Participatory design hence became one major set of tools within 
the toolbox of HCI. However, as Clay Spinuzzi argues, corporate participa- 
tory design reinforced a division of labour between designers and users that 
contradicted the emancipatory ideas of workplace democracy:”® One attempt, 
for instance, to treat user participation from an engineering perspective is the 
use of “personas”.?”® These are abstract archetypical members of the target 
group based on observation but defined by the designers. Personas, accord- 
ing to Kari Rönkkö, include the user’s perspective into the development by at 
the same time excluding the user from major parts of the work.*° 

At this point Peter Denning and Pamela Dargan draw a general line 
through the HCI field dividing it into two parts: On the one hand they see 
engineering approaches, including software engineering and the cognitive 
science inspired work for which design leads to products that fulfil specifica- 
tions. On the other hand there are “user-centred approaches” focusing on 
situated action and yielding satisfied customers.?! 


4.1.3 Getting inspiration: Artistic practices 


While one may follow Denning and Dargan and argue that the design 
of computer systems evolved between engineering and user-centred design, 
other ideas developed, too. HCI has repeatedly been seen as a field in which 
spaces of action and articulation are defined because design defines “the 
space of what can be said.”*? With the interface seen as a social and therewith 
relational space, the discourse opened for inputs from a new field: artistic 
practices were taken up, and in opposition to the “engineer-designer” the idea 
of the “artist-designer” was promoted.” At the same time, large parts of the 
HCI community attempted to change the name of the field into “interaction 
design”**. 

In a discourse that already treated interfaces as spaces for communi- 
cation, ideas from art forms that had already concentrated on space and 
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communication could have a major influence. The filter through which these 
ideas entered the HCI discourse again was space: within a project that aimed 
to develop interfaces for urban and rural public space in Italy, Norway and the 
Netherlands, interventionist strategies were applied to the interface design 
problem.°® Artistic maps, disposable photo cameras, albums and postcards 
were given to the target group (elderly people in this case) and understood as 
“cultural probes” that were to create visual and textual response. From such 
subjectively sampled material, prototypes and situations were constructed 
and again confronted with the audience. Finally, interventions that artisti- 
cally treated the results of the process were suggested. 

The official genealogy of Gaver and Dunner positions this approach in a 
line with situationism (especially, of course, psychogeography) answering the 
questions about the situated use of new artefacts with situationist strate- 
gies. However, cultural probes implicitly recalled some participatory design 
projects in which already methods such as end-user photography and story- 
telling had been applied.*° 

For the HCI field the probes promised to be able to answer a fundamental 
question that neither engineering nor user-centred design approaches could 
handle. Namely, what applications should be developed for novel technolo- 
gies? This is a question that historically proved hard to handle because on 
the one hand users tend to use new technologies in an unintended way, and 
on the other hand users in participatory design sessions are likely to suggest 
applications they already are familiar with. The HCI practice hence depends 
on methods to “identify needs” of prospective users?” and to forecast their use 
of what is offered them. One method to deal with this problem by studying 
such “possible situations of use”** has been developed in marketing research. 
It consists of a marketing analysis focusing on “lead users” — avant-garde 
users that are already familiar with new technologies and that therefore may 
serve as a “need-forecasting laboratory.” 

With similar goals the cultural probes approach was taken up by other 
researchers from the HCI field. In order to not only probe the context of the 
intended users but to directly involve them in the design of new technologies, 
the concept was extended creating “technology probes.” These, as opposed 
to non-functional mockups, are functional prototypes using new technology 
and are deployed in real-world situations where they observe their own use.*° 
Hence ‘technology probes’ not only provide a common language enabling par- 
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ticipation, but also become active players in the language game they are part 
of. Remarkably, all the projects working with probes resulted in designs that 
primarily wanted to create possibilities of communication. 


4.2 How software makes a difference 


Software which operates in social contexts needs continual management 
during its whole life cycle. Software engineers have simultaneously to track 
and reconstruct the social context upon which the software is acting. That 
is the only way to make sure it keeps its validity and does not progressively 
lose acceptance (see Laws of Software Evolution below). This requirement is 
due to the fact that software changes its own operation domain and causes 
a growing mismatch between the model of its domain, which is implemented 
in the software, and the domain itself. From a general perspective we have 
different agents acting in the same space, such as individuals, communi- 
ties, institutions, texts, concepts, programs, buildings, and other sorts of 
artefacts that mutually influence each other. From the Actor-Network-Theory 
of Bruno Latour (and others) we know that in such networks humans are 
not the only ones who act. According to its theory, the Actor-Network links 
together elements (and all the elements of successive links) to a network 
of mutual influence that performs as a whole. The Actor-Network-Theory, 
which treats humans and artefacts symmetrically, tries thereby to explain 
how material-semiotic networks generate certain results and behaviour as 
an integral entity. One interesting aspect about this theory is that it rejects 
the naive view of technical artefacts or humans existing prior and independ- 
ently of their participation in the social and semiotic network of interactions. 
Accepting this line of argumentation, we also have to regard software as an 
acting entity within a complex network. In general, software is linked to many 
other agents, thus it becomes effective on mainly two levels:*! 


- mental: programs objectify abstract ideas. They implement concepts 
that become part of daily life. The users are forced, whether they want 
or not, to deal with this new reality, interpret it and integrate it in their 
own subjective way into their view of the world; 

- auto operational: as self-executing entities, programs have an instan- 
taneous effect on reality. For example, robots, via their effectors, can 
directly manipulate their own environment. But such self-operating 
systems in general, if they are working in a social context, change 
the conditions of their operation since they open and close options for 
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alternative choices. They are directly connected to the action space of 
the participating users and actively alter it. 


4.2.1 Models and E-type applications 


Models are of central importance in the whole field of computer science. 
Consequently, during the common software development process different 
sorts of models are also developed and deployed. To achieve a deeper under- 
standing of the complex connections between software and its application 
domain, we have to briefly inspect modelling theory. The key properties of the 
general concept of a model are according to:*” 


Mapping property 

Models are always models of something, namely mappings, i.e. rep- 
resentations of natural or artificial originals, which themselves might 
be models. The concept of a mapping coincides with the concept of 
assigning model attributes to original attributes. That means mapping 
is based on the mathematical definition of mappings; 


Reduction property 

Models in general never capture all attributes of the original they rep- 
resent, but only those which seem to be relevant for the creator or user 
of the model; 


Pragmatic property 

Models are not per se unambiguously assigned to their originals. They 
fulfil their function of substitution a) for specific — recognising, and/ 
or acting, model-using — subjects, b) within a certain time interval c) 
under the restriction to certain mental or actual operations. 


According to Meir M. Lehman, under the perspective of modelling, a pro- 
gram “is a model of a model within a theory of a model of an abstraction of a 
portion of the world or of some universe of discourse.”** This quote points out 
that in software engineering at least three models are at work: the application 
model (including the domain and context), the formal model (specification), 
and finally the program itself.** Lehman’s early work on software engineer- 
ing and the growth dynamics of programs started from the conviction that 


Software Evolution is intrinsic to large systems. During his long-standing 
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work it became more and more clear that the concept of largeness could not 
provide the suitable basis for the study of Software Evolution. To solve this 
problem a new software classification scheme was proposed that was not 
based on size. Programs were divided into types S, P and E, whereby the most 
important class in the present context are the E-type applications. 

A program is called an S-type application, if the necessary and sufficient 
condition for the acceptance and proof of success of the whole development 
is its correctness in the full mathematical sense, relative to a given formal 
specification. It is assumed that the specification can be fully predetermined 
before the development of the software begins. The computation of math- 
ematical functions and/or all sorts of formally definable transformations (e.g. 
compilers or proof procedures), are typical examples of S-type applications. 

The E-type applications are closely related to the concept called Software 
Evolution. In contrast to the S-Type, E-programs are applications which 
become part of the domain they are modelling. That means that they are not 
only embedded into the real world but change the reality they are living in 
through their usage and activity. These programs become effective in reality, 
they modify what they model. Thus, E-Type programs are the ones we have to 
deal with in participatory media development. 

Somehow halfway between S-type and E-type programs the P-type is 
located. These programs address problems that are fully specifiable, but the 
results of the execution has to be checked against the application domain 
and not its specification model. A typical example might be weather forecast- 
ing systems. They are mathematically correctly describable but their per- 
formance has to be demonstrated in reality, i.e. in comparison to the actual 
weather. On the other hand, the execution of the program does not change its 
domain, e.g. influence the weather, like E-type programs do. 


4.2.2 Software Evolution and some of its laws 


In its most general meaning the term evolution tries to capture the phe- 
nomenon of progressive change of the attributes of a system. From this point 
of view, not only nature evolves over time but also cities, societies, ideas, 
theories and also software. Of course, for all these different types of evolution 
we have to say what progress means in a particular field of evolution. The 
term evolution in software engineering is used to describe the cyclic and con- 
tinual structure of maintenance, after the initial development phase. But it is 
important to mention that the meaning of maintenance in software engineer- 
ing is incompatible with its common usage. Lehman and Fernändez-Ramil 
point out that over the years it has been recognised that the term has to be 
carefully used in the context of software development. Normally, the term 
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describes the efforts of people to soften or neutralise the processes of ageing, 
wear and tear or other forms of deterioration, which are typical and ongoing 
in every material artefact. For software maintenance it is more about the rec- 
tification of assumptions made about the application. We do not have to care 
about the ageing material but about the ageing of our thoughts, expectations, 
and models. “What happens with software is that it is changed or adapted to 
maintain it satisfactorily in changed domains and under new circumstances 
as judged by stakeholders such as users. Software is evolved to maintain 
embedded assumptions and its compatibility valid with respect to the world 
as it is now. Only in this sense, is the use of the term ‘maintenance’ appropri- 
ate in the software context.”*> 

The research on Software Evolution came up with a set of behaviour that 
is now known as Lehman’s laws. Although these laws mainly apply to mono- 
lithic, proprietary software systems, some of the most important laws are 
obviously also valid for participatory design approaches or open source soft- 
ware developments. In the following, we do not present a complete list of all 
laws of Software Evolution but just a few that seem most appropriate within 


our participatory Unortkataster development:*° 


Continuing Change: An E-type program that is used must be continu- 
ally adapted or else it becomes progressively less satisfactory; 
Continuing Growth: Functional content of a program must be continu- 
ally increased to maintain user satisfaction over its lifetime; 

Declining Quality: E-type programs will be perceived as of declining 
quality unless rigorously maintained and adapted to a changing opera- 
tional environment; 

Feedback System: E-type programming processes constitute Multi- 
loop, Multi-level Feedback systems and must be treated as such to be 


successfully modified or improved. 


5. Unortkataster: An urban experiment 


The development of the Unortkataster Köln is based on cooperation with a 
group of lead users. The working group Bild der Stadt of the initiative Leitbild 
2020 formulated the idea of creating a technological platform for the citizens 
of Cologne to work on the city, from within the city. The continual talks with 
members of the community are the basis for an iterative software develop- 


ment process. 
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The Unortkataster application is an outcome of this collaboration and pro- 
vides a community platform for the citizens of Cologne, based on a shared 
city-map. Adding state-of-the-art community functionalities to a digital map 
extends it to a multi-user interface with user administration capabilities and 
a database structure for storing media content. The map thereby allows peo- 
ple to mark areas or places on the map that are considered to be an Unort, 
and to add opinions and media content to justify the personal decision for the 
marking. What happens if maps become software interfaces? What does the 
term Unort suggest and how is the added content related to the meaning and 
function of the administrative Kataster maps? 

On the Unortkataster, marking space becomes equal to starting and locat- 
ing a discussion about that special location. The setting of a marker creates 
an Unort and is the start of a discussion about the place. Other people may, 
in succession, add comments or media content to express affirmation or dis- 
affirmation. The Unortkataster is intended to facilitate processes, therefore 
time becomes the second principle of organisation. Since each action on the 
map is recorded by time, the development of the discussion can be regarded 
relative to space. It hence constitutes a spatio-temporal interface through 
which audiovisual communication is made possible. Conversations are con- 
nected with places, hence places become blogs. This allows subscribing to the 
information feed of a place and the tracking of the ongoing discussion. Places 
have a start and possible end in time. They may appear and vanish. If citizens 
decide to take part in the discussion about places in Cologne, the time layer 
of the map is updated and thereby the discussion is documented. 


5.1 Maps and software 


The originator of the General Semantics discipline, Alfred Korzybski 
stated, “A map is not the territory it represents, but if correct, it has a similar 
structure to the territory, which accounts for its usefulness.” This points to 
what we have said about models and modelling. Our perception of reality is 
not reality itself but just a subjective version, or our own map that fulfils a 
certain function within a certain time interval and under the restriction of 
certain mental or actual operations. 

Maps are figurative representations of relations or objects in the physi- 
cal or logical world. They represent a certain selection that is related to the 
observation of an author. Maps contain hierarchies that influence how we see 
the world and are often based on arbitrary conventions. In the history of map- 
making, the setting of conventions was part of the legitimisation of power. If 
map users become mapmakers, they are empowered to set the conventions 
themselves and define their own territories. Software driven mapping sys- 


209 


tems extend maps into dynamic structures. When maps become digital, they 
cut the line between the database and the image. The map becomes an inter- 
face that allows communication with the database. It thereby structures the 
circulation of information as an interaction of command, addressing, dating, 
storing and feedback. 

Maps have to be upgraded constantly if they relate to a dynamic content. 
In the Unortkataster, the updating of the map is conceded to the partici- 
pants. They start building the dynamic map upon the static map delivered 
by Google. Thus, the map’s view evolves from a single-sided perspective to a 
many-sided perspective on the city. The significance of observation by users 
acquires another reputation. 


5.2 Orte and Unorte 


In order to analyse the phrasing of Unort, the meaning of Ort (place) is to 
be considered. Martina Löw states that places become visible as the target 
and result of the placing of social goods or people. Löw notes that places 
emerge by placements. In order to enable things to be placed, to be able to 
place something, space has to exist. On the other hand, the process of plac- 
ing leads to spacing. The characteristics of places in physical space can be 
compared to those in an online environment: places appear by placings but 
are not identical with them.* On an online map, a marked place may relate 
to a real place in the city but, at the same time, generate a different kind of 
place in the online setting. Marking an Unort in this context leads to another 
paradox: when people mark places on the Unortkataster-map this becomes 
an Unort but at the same time it is becoming an Ort. The indissoluble contra- 
diction of the two terms is productive as it maintains controversy about the 
place - being an Unort or not - alive. The Unort may thus become a catalyst for 
social controversies about places in the city. It is thereby to be distinguished 
from the Non-Place that Marc Augé describes as a space where neither identi- 
ties nor relations to history are legible.** 

Marking on the application happens in a virtual environment. The ques- 
tion of which resulting changes may happen in physical space is, in con- 
trast, related to administrative structures and political processes. But the 
Unortkataster delivers the possibility for a controversy in a setting where a 
single person is no longer authorised to change an environment, i.e. in public 
space. Public space, in this sense, is defined as the area that is exposed to the 
criticism of the public audience. It stands in opposition to private places and 
exclusive territories that allow an owner or a group of owners to exclude criti- 
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Fig. 2. Unorte in Cologne (Top left to bottom right): Building site close to the 
Dom, chewing gum, messy glass containers, dirty corner behind the station 
(Authors from top left to bottom right: Stefan Göllner, Keiko Takahashi, Jan Hopmann, 
Oliver Salkic) 


A 


cism to a certain degree. Thus, questions which are related to the relations of 
public and private properties become of major import for the platform. 

Individual observation, marking places and the interlinking of different 
opinions, are the basic actions of participating in the Unortkataster. The 
tool thereby acts as a platform for organising this communication process. 
As the object of observation happens to be the public sphere, any favoured 
change will target the exponents of the public, which are represented firstly 
by the city administration. The Unortkataster thus provides insight into how 
citizens observe the city. The administration is excluded from this process 
but will be put in the position to follow the ongoing discussion and might 
draw conclusions out of this. Conflicts among users, property owners and 
city administration may arise because of activities on the platform. On that 
score, the setup of effective moderation mechanisms will be a crucial part of 
the research of the project. 
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5.3 Kataster leads to Unortkataster 


The administrative Kataster-map mainly intends to deliver an up-to- 
date data basis for the land register and to protect private property rights. 
Although the Kataster is used for a completely different purpose than the 
Unortkataster, there are some crucial similarities between the two different 
kinds of maps. The Unortkataster does not target on archiving of facts and 
conditions. The content of the application therefore is soft: contributions and 
related media articulate opinions of participants that might be put into per- 
spective by any kind of reply or addition of another user. The dynamics of 
an Unort lies in the impossibility of its final definition. The negative rating 
that distinguishes the Unort from the Ort refers to the personal opinion of its 
author and all conclusions drawn from it by related critics and supporters. 
The value of an Unort for the application is measured by the participants’ 
interest in adding their own contributions. Both Kataster-maps have in com- 
mon open-endedness and dependence on the permanent input of citizens. 
The Kataster documents evolution of private property and is thereby related 
to architecture and the built structure. The Unortkataster documents social 
dynamics that proceed in relation to the public city stage. When input stops, 
both applications become historic documents that demonstrate how the 


social sphere and built structure changed over time. 


5.4 Unortkataster as a web-probe 


Web 2.0 applications have recently shifted the focus away from the user 
and toward the community. With that shift, methods such as the persona 
that centre around the single user, its needs and its coupling to the interface, 
have become much less appropriate. Instead here, early probing implicitly 
developed towards a standard design method. Within this setting (and with- 
out explicit reference to the participatory design tradition), this methodology 
has been called “perpetual beta”*: Applications are already opened for use 
during development, while their development cycle is possibly never declared 
finished: a praxis that, in the famous example of flickr, yielded a success- 
ful photo sharing application that had originally been developed (now aban- 
doned) as an online game. 

Methodologically, the Unortkataster developed through a crossover of 
practices described above. It started from a lead user approach while the 
Unortkataster itself may be seen as a Web 2.0 probe. The concept of the Unort 
was (and continues to be) discussed by probing the city through various media. 
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As an E-type applica- 


wnortkataster jog 1011x786) Pixel 


tion it borrows various 
ideas and methods from 
the history of software 
development, while at 
the same time neither 
a product nor a satis- 
fied customer are its 
intended outcome. At 
best, it will generate dis- 
course, structured by 
a dynamical (software 
based) map of the city, 


acting back upon the 


city and ats ical plages Fig. 3. Unortkataster application prototype 


through the discourse it developed together with lead users of the initia- 
generates. Similar to the tive »Leitbild Köln 2020« 

prototypes of the partici- eee 
patory design tradition, 

it is intended to create a common language for discussing the city seen as 
a space of shared problems. As a technology probe it is structuring this dis- 
course, at the same time it is open to being shaped by it. Moreover, the data- 
base on which the Unortkataster is built may also have mobile access. How 
do Unorte change when they can be created and discussed in situ? How does 
perception of the city change, viewed through a mobile representation of the 
Unorte? 


6. Conclusion 


Participatory media development is not least about balances of power. In 
the end the forces and interests of the people driving the development of a 
certain application have to aid one another instead of blocking each other. An 
advised motivation, moderation and guidance approach seems to be the key 
to a successful collaborative development. Direction of the development and 
at the same time the balancing of power between the participants is subject 
to the new engineer-sociologist, it is actually his major power. Today we have 
to accept that there is no longer any technical activity that is not, at the same 
time, a cultural activity, and vice versa. 

According to Wikipedia, collaborative governance is a process and a form 
of governance in which participants (parties, agencies, stakeholders) repre- 
senting different interests are collectively empowered to make a policy deci- 
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sion or make recommendations to a final decision-maker who will not sub- 
stantially change consensus recommendations from the group. It is obvious 
that not all urban problems can be solved with participatory media on the 
basis of some sort of communication in the wild. For example, serious resource 
allocation conflicts (were does the money, manpower, engagement, etc. of the 
city go) also need intervention of a powerful government. Without that, the 
new forms of cooperation would mainly be “a new realm of creative expres- 
sion and empowerment for the middle classes while the interests of the less 
powerful will continue to be represented (and distorted) by the devalued tra- 


ditional welfare structures of (local) government. ”®? 
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1. Introduction 
This chapter presents research on the modelling of expressive gesture in 
multimodal interaction and on the development of multimodal interactive 
3 
systems explicitly taking into account the role of expressive gesture in the $ 
communication process. In this perspective, a particular focus is on dance 
and music performances as first-class conveyors of expressive and emotional g 
S 
content. s 
Expressive gesture is a key concept in our research.! This paper tries to er 
SE 
K 


deal with it, and introduces two experiments aiming at understanding the 
non-verbal mechanisms of expressive /emotional communication. 

Several definitions of gesture exist in the literature. The most common 
use of the term is with respect to natural gesture, which is defined as a sup- 
port to verbal communication. For Cassel and colleagues (1990) “[a] natural 
gesture means the types of gestures spontaneously generated by a person 
telling a story, speaking in public, or holding a conversation.” McNeill (1992) 
in his well-known taxonomy divides the natural gestures generated during 
a discourse into four different categories: iconic, metaphoric, deictic, and 
beats. In a wider perspective, Kurtenbach and Hulteen (1990) define gesture 
as “a movement of the body that contains information.”? 

In artistic contexts, and in particular in the field of performing arts, ges- 
tures are often not intended to denote things or to support speech as in the 
traditional framework of natural gesture, but the information they contain 
and convey is related to the affective/emotional domain. From this point of 
view, gestures can be considered “expressive” depending on the kind of infor- 
mation they convey: expressive gestures carry what Cowie et al. (2001) call 
“implicit messages”, and what Hashimoto (1997) calls KANSEI. That is, they 
are responsible of the communication of a kind of information (what we call 
expressive content) which is different and in most cases independent from, 
even if often superimposed on, a possible denotative meaning, and which con- 
cerns aspects related to feelings, moods, affect, and emotional intentions. 

For example, the same action can be performed in several ways, by stress- 
ing different qualities of movement: it is possible to recognise a person from 
the way she or he walks, but it is also possible to obtain information about 
the emotional state of a person by looking at her or his gait, e.g., if she or he 
is angry, sad, happy. In the case of gait analysis, we can therefore distinguish 
among several objectives and layers of analysis: a first one aiming at describ- 
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2 A survey and a discussion of existing definition of gesture can be found in Cadoz 
and Wanderley 2000. 
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ing the physical features of the movement, for example in order to classify 
it; a second one aiming at extracting the expressive content gait conveys, 
e.g., in terms of information about the emotional state that the walker com- 
municates through her or his way of walking. From this point of view, walking 
can be considered as an expressive gesture: even if no denotative meaning 
is associated with it, it still communicates information about the emotional 
state of the walker, i.e., it conveys a specific expressive content. In fact, in this 
perspective the walking action fully satisfies the conditions stated in the defi- 
nition of gesture by Kurtenbach and Hulteen (1990): walking is “a movement 
of the body that contains information.” Some studies can be found aiming 
at analysing the expressive intentions conveyed through everyday actions: 
for example, Pollick (2001) investigated the expressive content of actions like 
knocking or drinking. 

If on the one hand expressive gestures partially include natural gestures, 
that is, natural gestures can also be expressive gestures, we face on the other 
hand a more general concept of expressive gesture that includes not only nat- 
ural gestures but also musical, human movement, and visual (e.g. computer- 
animated) gestures. Our concept of expressive gesture is therefore somewhat 
broader than the concept of gesture as defined by Kurtenbach and Hulteen, 
since it also considers cases in which, with the aid of technology, communica- 
tion of expressive content takes place even without an explicit movement of 
the body, or, at least, the movement of the body is only indirectly involved in 
the communication process. This can happen, for example, when using visual 
media. The expressive content is conveyed through a continuum of possible 
ways ranging from realistic to abstract images and effects: cinematography, 
cartoons, virtual environments with computer-animated characters and ava- 
tars, and expressive control of lights in the context of a theatre (e.g. related 
to actor’s physical gestures). Consider, for example, a theatre performance: 
the director, choreographer, composer can ask actors, dancers, musicians, to 
communicate content through a number of expressive gestures (e.g., dance 
and/or music phrases). At the same time, technology allows the director to 
extend the language available to him. He can map motion or music features 
onto particular configurations of lights, in movements of virtual characters, 
in automatically generated computer music and live electronics. In this way, 
he can create an “extended” expressive gesture that, while still having the 
purpose of communicating an expressive content, is only partially related to 
explicit body movements: in a way, such “extended expressive gesture” is the 
result of a juxtaposition of several dance, music, and visual gestures, but it 
is not just the sum of them, since it also includes the artistic point of view 


nn 
4 Quite a lot of research work can be found in the computer vision literature about 
gait analysis, see for example Liu et al. 2002. 
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of the director who created it, and it is perceived as multimodal stimuli by 
human spectators. 

Our research on expressive gesture aims at the development of interac- 
tive multimedia systems enabling novel interaction paradigms and allowing 
a deeper engagement of the user by explicitly observing and processing his/ 
her expressive gestures. Since artistic performances use non-verbal commu- 
nication mechanisms to convey expressive content elaborately, we focused 
on performing arts, and in particular on dance and music performances, as 
a test-bed where computational models of expressive gesture and algorithms 
for expressive gesture processing can be developed, studied, and tested. 

In particular, our attention has been focused on two aspects: 


- Expressive gesture as a way to convey a particular emotion to the 
audience; 


- Expressive gesture as a way to emotionally engage the audience. 


Each of these has recently been the subject of experiments at our lab aim- 
ing at understanding which features in an expressive gesture are responsible 
for the communication of the expressive content, and how the dynamics of 
these features correlates with a specific expressive content. 

In this paper, we concretely illustrate our approach by presenting two 
experiments focused on these two aspects. 

The first one aims at (i) individuating which motion cues are mostly 
involved in conveying the dancer’s expressive intentions (in term of basic 
emotions) to the audience during a dance performance and (ii) testing the 
models and algorithms developed by comparing their performances with 
spectators’ ratings of the same dance fragments. 

The second one investigates the mechanisms responsible for the audi- 
ence’s engagement in a musical performance. The aim of this experiment 
is again twofold: (i) individuating which auditory and visual cues are most 
involved in conveying the performer’s expressive intentions and (ii) testing the 
developed model by comparing their performances with spectators’ ratings of 
the same musical performances. 

For the analysis of expressive gesture in these experiments a unifying 
conceptual framework was adopted. 
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2. The Layered Conceptual Framework 


The experiments presented in this paper address expressive gesture in 
music and dance performance. 

While gesture in dance performance mainly concerns the visual/physical 
modality (even ifthe auditory components can be relevant as well), gesture in 
music performance uses both the auditory and the visual channels to convey 
expressive information, and, thus, it is multimodal in its essence. Gestures in 
music performance are not only the expressive and functional gestures that 
a performer physically makes, but also include expressive gestures present 
in the sound produced. When we define gestures as structural units that 
have internal consistency and are distinguished in time and quality from 
neighbouring units, it is possible to analyse gestures in both modalities. 
Multimodality is therefore a key issue. In order to deal with multimodal input 
a unifying conceptual framework has been adopted.? It is based on a layered 
approach ranging from low-level physical measures (e.g., position, speed, 
acceleration of body parts for dance gestures, sampled audio signals or MIDI 
messages for music gesture) toward descriptors of overall gesture features 
(e.g., motion fluency, directness, impulsiveness for dance gestures, analysis 
of melodic and harmonic qualities of a music phrase for music gestures). 

This layered approach is sketched in Figure 1. Each layer is depicted with 
its inputs, its outputs, and the kind of processing it is responsible for. In the 
following sections, each layer will be discussed with respect to its role in the 
two experiments. 

Our conceptual framework, here presented for analysis, can also be 
applied for synthesis of expressive gesture: for example for the generation 
and control of the movement of avatars, virtual characters, or robots in Mixed 
Reality scenarios, as well as for the synthesis and interpretation of music. 
Examples of synthesis of expressive movement and expressive audio content 
are well documented in literature.® 

Finally, it should be noticed that in the perspective of developing novel 
interactive multimedia systems for artistic applications, such a framework 
should be considered in the context of a broader Mixed Reality scenario in 
which virtual subjects (e.g., virtual characters) who behave both as observ- 
ers and as agents perform the four layers of processing in the analysis of 
observed expressive gestures and in the synthesis of expressive gestures to 
communicate (directly or remotely) with other real and virtual subjects. 
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6 See e.g. the EMOTE system (Chi et al. 2000) for generation of movement of avatars 
and virtual characters based on high level motion qualities, and the systems for synthe- 
sis of expressive music performances developed at KTH (Friberg et al. 2000) and by the 
DEI-CSC group at the University of Padova (Canazza et al. 2000). 
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High-level expressive information: (Experiment 1) Recognised emotions (e.g., anger, fear, grief, joy); 
(Experiment 2) Predict spectators' intensity of emotional experience. 


t 


Layer 4 - Concepts and structures 
Modelling techniques (for example, classification in terms of basic emotions, or prediction of intense 


emotional experience in spectators): e.g., based on multiple regression, neural networks, support 
vector machines, decision trees, Bayesian networks. 


Î 


Segmented gestures and related parameters (e.g., absolute and relative duratons), trajectories 
representing gestures in semantic spaces. 


i 


Layer 3 — Mid-level features and maps 
Techniques for gesture segmentation: motion segmentation (e.g., in pause and motion phases), 
segmentation of musical excerpts in musical phrases. Representation of gestures as trajectories in 
semantic spaces (e.g., Laban's Effort space, energy-articulation space) 


Î 


Motion and audio descriptors: e.g., amount of energy - loudness, amount of contraction/expansion - 
spectral width and melodic contour, low fluency - roughness etc. 


ft 


Layer 2 — Low-level features 
Computer vision techniques on the incoming images, statistical measures, signal processing 
techniques on audio signals. 


t 


- Images pre-processed to detect movement, trajectory of points (e.g., trajectories of body parts, 
trajectories of dancers in the space) 
- MIDI and audio pre-processed to detect spectral and temporal low-level features. 


Layer 1 — Physical signals 
Analysis of video and audio signals: techniques for background subtraction, motion detection, 


motion tracking (e.g., techniques for colour tracking, optical flow based feature tracking), techniques 
for audio pre-processing and filtering, signal conditioning. 


f 


Data from several kinds of sensors, e.g., images from video cameras, positions from localisation 
systems, data from accelerometers, sampled audio, MIDI messages. 


Fig. 1. The layered conceptual framework and its instantiation in the two 
experiments 
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3. Modeling expressive gesture in dancers 


As an example of analysis of expressive gesture in dance performance, 
we discuss an experiment carried out in collaboration with the Department 
of Psychology of the University of Uppsala (Sweden) in the EU-IST MEGA 
project. 

The aim of the experiment was twofold: (i) individuating which motion 
cues are mostly involved in conveying the dancer’s expressive intentions to 
the audience during a dance performance and (ii) testing the developed mod- 
els and algorithms by comparing their performances with spectators’ ratings 
of the same dance fragments. 

In the case of this experiment, expressive gesture was analysed with 
respect to its ability to convey emotions to the audience. The study focused 
on the communication through dance gesture and recognition by spectators 
of the four basic emotions: anger, fear, grief, and joy. 

The research hypotheses are grounded in the role of the Laban’s dimen- 
sions in dance gesture, as described in Laban’s Theory of Effort:” 


- The time dimension in terms of overall duration of time and tempo 
changes also elaborated as the underlying structure of rhythm and 
flow of the movement; 

- The space dimension in its aspects related to Laban’s “personal 
space” e.g., to what extent limbs are contracted or expanded in rela- 
tion to the body centre; 

- The flow dimension in terms of analysis of shapes of speed and energy 
curves, and frequency/rhythm of motion and pause phases; 

- The weight dimension in terms of amount of tension and dynamics in 


movement, e.g., vertical component of acceleration. 


These cues were predicted to be associated in different combinations to 


each emotion category.® 


3.1 The experiment 


An experienced choreographer was asked to design a choreography such 
that it excluded any propositional gesture or posture and it avoided stere- 


otyped emotions. 
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7 Laban 1963; Laban/Lawrence 1947 
8 Details can be found in Camurri/Lagerlöf/Volpe 2003. 
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In Uppsala, five dancers performed this same dance with four different 
emotional expressions: anger, fear, grief and joy. Each dancer performed all 
four emotions. The dance performances were video-recorded by two digital 
videocameras (DV recording format) standing fixed in the same frontal view 
of the dance (a spectator view). One camera obtained recordings to be used 
as stimuli for spectators’ ratings. The second video camera was placed in 
the same position but with specific recording conditions and hardware set- 
tings to simplify and optimise automated recognition of movement cues (e.g., 
high speed shutter). Dancers’ clothes were similar (dark), contrasting with 
the white background, in an empty performance space without any scenery. 
Digitised fading eliminated facial information and the dancers appeared as 
dark and distant figures against a white background. 

The psychologists in Uppsala then proceeded in collecting spectators’ rat- 
ings: the dances were judged with regard to perceived emotion by 32 observ- 
ers, divided in two groups. In one group ratings were collected by ‘forced 
choice’ (choose one emotion category and rate its intensity) for each perform- 
ance, while the other group was instructed to use a multiple choice schema, 
i.e., to rate the intensity of each emotion on all four emotion scales for each 
performance. 

At the same time, at the InfoMus Lab we proceeded in extracting motion 
cues from the video recordings and in developing models for automatic clas- 
sification of dance gestures in terms of the basic emotion conveyed. 


3.2 Automated Extraction of Motion Cues 


Extraction of motion cues followed the conceptual framework described 
in Section 2. 


3.2.1 Layer 1 


In the case of analysis of dance performance from video, layer 1 is respon- 
sible for the processing of the incoming video frames in order to detect and 
obtain information about the motion that is actually occurring. It receives as 
input images from one or more videocameras and, if available, information 
from other sensors (e.g., accelerometers). Two types of output are generated: 
processed images and trajectories of body parts. Layer 1 accomplishes its 
task by means of consolidated computer vision techniques usually employed 
for real-time analysis and recognition of human motion and activity.’ It 
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9 See for example the temporal templates technique for representation and recogni- 


tion of human movement described in Bobick/Davis 2001. 


should be noted that in contrast to the research of Bobick and J. Davis, we 
do not aim at detecting or recognising a specific kind of motion or activity. The 
techniques we use include feature-tracking based on the Lucas-Kanade algo- 
rithm”, skin colour tracking to extract positions and trajectories of hands 
and head, an algorithm to divide a body silhouette into sub-regions, and 
Silhouette Motion Images (SMIs). A SMI is an image carrying information 
about variations of the silhouette shape and position in the last few frames. 
SMIs are inspired by motion-energy images (MEI) and motion-history images 
(MHI).!! They differ from MEIs in the fact that the silhouette in the last (more 
recent) frame is removed from the output image: in such a way only motion 
is considered, while the current posture is skipped. Thus, SMIs can be con- 
sidered as carrying information about the “amount of motion” which has 
occurred in the last frames. Information about time is implicit in SMI and is 
not explicitly recorded. We also use an extension of SMIs, which takes into 
account the internal motion in silhouettes. In such a way we are able to dis- 
tinguish between global movements of the whole body in the General Space 
and internal movements of body limbs inside the Kinesphere. 


3.2.2 Layer 2 


Layer 2 is responsible for the extraction of a set of motion cues from 
the data coming from low-level motion tracking. Its inputs are the processed 
images and the trajectories of points (motion trajectories) coming from Layer 
1. Its output is a collection of motion cues describing movement and its 
qualities. According to the research hypotheses described above, the cues 
extracted for this experiment include: 


- Cues related to the amount of movement (energy) and in particular 
what we call Quantity of Motion (QoM). QoM is computed as the area 
i.e., number of pixels) of an SMI.!? It can be considered as an over- 
all measure of the amount of detected motion, involving velocity and 
force; 

- Cues related to body contraction/expansion, and in particular the 
Contraction Index (CI). CI is a measure, ranging from 0 to 1, of how the 
dancer’s body uses the space surrounding it. The algorithm to com- 
pute the CI’* combines two different techniques: the individuation of 
an ellipse approximating the body silhouette and computations based 
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10 Lucas/Kanade 1981 

11 Bradsky/Davis 2002; Bobick/Davis 2001 
12 Camurri/Lagerlöf/Volpe 2003 

13 Camurri/Lagerl6f/Volpe 2003 
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on the bounding region. The former is based on an analogy between 
the image moments and mechanical moments:!* the eccentricity of the 
approximating ellipse is related to body contraction/expansion. The 
latter compares the area covered by the minimum rectangle surround- 
ing the dancer with the area currently covered by the silhouette; 

- Cues derived from psychological studies'® such as amount of upward 
movement, dynamics of the Contraction Index (i.e., how much CI was 
over a given threshold along a time unit); 

- Cues related to the use of space: length and overall direction of motion 
trajectories; 

- Kinematical cues (e.g., velocity and acceleration) calculated on motion 
trajectories. 


For those cues depending on motion trajectories a Lucas-Kanade feature 
tracker was employed in Layer 1. A redundant set of 40 points randomly 
distributed on the whole body was tracked. Points were reassigned each time 
dancers stopped their motion (i.e., a pause was detected) so that a small and 
non-significant amount of points was lost during tracking. Overall motion 
cues were calculated by averaging the values obtained for each trajectory. 


3.2.3 Layer 3 


Layer 3 is in charge of segmenting motion in order to individuate motion 
and non-motion (pause) phases. QoM was used to perform such segmenta- 
tion. QoM is related to the overall amount of motion and its evolution in time 
can be seen as a sequence of bell-shaped curves (motion bells). In order to 
segment motion, a list of these motion bells was extracted and their features 
(e.g., peak value and duration) computed. An empirical threshold was defined 
for these experiments: the dancer is considered to be moving if its current 
QoM is above 2.5% of the average value of the QoM computed along each 
whole dance fragment. 

Segmentation allows further higher-level cues to be extracted, e.g., cues 
related to the time duration of motion and pause phases. A concrete example 
is the Directness Index (DI), calculated as the ratio between the length of the 
straight trajectory connecting the first and the last point of a motion trajec- 
tory and the sum of the lengths of each segment constituting the trajectory. 
Moreover, segmentation can be considered as a first step toward the analysis 
of the rhythmic aspects of the dance. Analysis of the sequence of pause and 
motion phases and their relative time durations can lead to a first evaluation 
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15 See for example Boone/Cunningham 1998 
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of dance tempo and its evolution in time, i.e., tempo changes and articula- 
tion (the analogue to music legato/staccato). Parameters from pause phases 
can also be extracted to differentiate real standing-still positions from active 
pauses involving low-motion (hesitation, subtle swaying or tremble, e.g., due 
to instable equilibrium or fatigue). 

Furthermore, motion fluency and impulsiveness can be evaluated. They 
are related to Laban’s Flow and Time axes. Fluency can be estimated start- 
ing from an analysis of the temporal sequence of motion bells. A dance frag- 
ment performed with frequent stops and restarts (i.e., characterised by a high 
number of short pause and motion phases) will gain the result of being less 
fluent than the same movement performed in a continuous, “harmonic” way 
(i.e., with a few long motion phases). The hesitating, bounded performance 
will be characterised by a higher percentage of acceleration and deceleration 
in the time unit (due to the frequent stops and restarts), a parameter that has 
been demonstrated to be of relevant importance in motion flow evaluation.!® 

A first measure of impulsiveness can be obtained from the shape of a 
motion bell. In fact, since QoM is directly related to the amount of movement 
detected, a short motion bell having a high peak value will be the result of an 
impulsive movement (i.e., a movement in which speed rapidly moves from a 
value near or equal to zero, to a peak and back to zero). On the other hand, 
a sustained, continuous movement will show a motion bell characterised by 
a relatively long time period in which the QoM values have little fluctuations 
around the average value (i.e., the speed is more or less constant during the 
movement). 

Fluency and impulsiveness are also related to the spectral content of the 
QoM: a movement having significant energy at high frequencies is a candi- 
date to be characterised by low fluency. 


3.2.4 Layer 4 


In this experiment, Layer 4 collects inputs from Layers 2 and 3 (18 varia- 
bles have been calculated on each detected motion phase) and tries to classify 
a motion phase in terms of the four basic emotions anger, fear, grief and joy. 

As a first step, statistical techniques have been used for a preliminary 
analysis: descriptive statistics and a one-way ANOVA have been computed 
for each motion cue." 


Decca 
16 See for example Zhao 2001, where a neural network is used to evaluate Laban’s 
flow dimension. 


17 Results of such preliminary analysis can be found in Mazzarino 2002; Camurri/ 
Lagerlöf/Volpe 2003; Volpe 2003. 
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Decision tree models were then built for classification. Five training sets 
(85% of the available data) and five test sets (15% of the available data) were 
extracted from the data set. The samples for the test sets were uniformly dis- 
tributed along the four classes and the five dancers. Five decision trees were 
built on the five training sets and evaluated on the five test sets. The Gini’s 
index of heterogeneity was used for building the decision trees. Decision trees 
were selected for this study since they produce rules that can be used to 
interpret the results. Comparison with other classification techniques (e.g., 
Neural Networks, Support Vector Machines) remains a task for possible 
future work. 

The above-described techniques in the four layers were implemented in 
our EyesWeb open software platform.'® The Expressive Gesture Processing 
Library’? includes these and other processing modules. 


3.3 Results 


Results from spectators’ ratings are described in Camurri/Lagerlöf/Volpe 
2003. The results obtained on the five decision trees can be summarised 
as follows (results for the best model are reported in Tables 1 and 2 (see p. 
232) showing the confusion matrices for the training set and for the test set 
respectively). 

Two models (3 and 5) fit the data set quite well; the rates of correct clas- 
sification on the training set for these two models averaged over the four 
classes are 78.5% and 61.6%, respectively. Three models (1, 2, and 4) have 
difficulties in classifying fear. The rates of correct classification on the training 
set for these three models averaged over the four classes are 41.9%, 38.7%, 
and 36.0%, respectively. Models 2 and 4 also have problems with joy, which 
means that they distinguish correctly only between anger and grief. 

A similar situation can be observed in the evaluation carried out on the 
test set: only models 3 and 5 are able to classify all four emotions correctly. 
Model 1 cannot classify fear, while models 2 and 4 cannot classify fear and 
joy. 

The rates of correct classification on the test set for the five models aver- 
aged on the four classes are respectively: 40%, 36%, 36%, 26%, and 40%. 
Thus the average rate of correct classification on the five models is 35.6%. 
Except for model 4, they are all above chance level (25%). Model 5 can be 
considered as the best model, since it has a rate of correct classification of 
40% and is able to classify all four emotions. 


ELLE 
18 Camurri et al. 2000; Free download of technical documentation and full software 
environment are available at <http:/ /www.eyesweb.org>. 
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Table 1. Confusion matrix for the training set for the best decision tree 
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Table 2. Confusion matrix for the test set for the best decision tree 
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These rates of correct classification, which at first glance seem to be quite 
low (40% being the best model), should however be considered with respect 
to the rates of correct classification by spectators who were asked to classify 
the same dances. In fact, spectators’ ratings collected by psychologists in 
Uppsala show a rate of correct classification (averaged over the 20 dances) 
of 56%. 

The rate of correct automatic classification (35.6%) is thus in between 
chance level (25%) and the rate of correct classification for human observers 
(56%). 

Furthermore, if the rate of correct classification for human observers is 
considered as a reference, and percentages are recalculated taking it as 100% 
(i.e., relative instead of absolute rates are computed), the average rate of cor- 
rect automatic classification with respect to spectators is 63.6%, and the best 
model (i.e., model 5) obtains a rate of correct classification of 71.4%. 

By observing the confusion matrix of the best model (both for the test set 
and for the training set) it can be noticed that fear is often classified as anger. 
This particularly holds for the test set, where fear is the basic emotion which 
receives the lowest rate of correct classification, since 6 of the 13 motion 
phases extracted from fear performances are classified as anger. Something 
similar can be observed in spectators’ ratings.”° 
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A deeper comparison with spectator’s ratings shows that while anger 
is generally well classified both by spectators and by the automatic system 
(60% for automatic recognition vs. 60.6% for spectators), quite bad results 
are obtained for fear (below chance level for the automatic classification). 
The biggest overall difference between spectators and automatic classification 
was observed for joy (70.4% for spectators vs. 27.7%, just above chance level, 
for automatic classification). In the case of grief instead, automatic classifica- 
tion performs better than human observers (48.3% for automatic classifica- 
tion vs. 39.8% for spectators): this happens in five cases and mainly for grief. 
In seven cases, the rate of correct classification for the automatic system is 
below chance level (and this always happens for fear). In one case, automatic 
classification did not succeed in finding the correct emotion (Fear — Dancer 
4), but spectators obtained 67% of correct classification. In another case, 
spectators’ ratings are below chance level (Grief - Dancer 5), but automatic 
classification could obtain a rate of correct classification up to 50%. 

Dancer 1 obtained the lowest rates of correct classification both from 
spectators and from the models. Dancer 5 obtains similar rates from both. 
Dancer 2 is the best classified by spectators and also obtains a quite high 
rate (with respect to the other dancers) in automatic classification. 


4. Analysis of expressive 
gesture in music performance 


The second experiment investigates the mechanisms responsible for the 
audience’s engagement in a musical performance.”! The aim of this experi- 
ment is again twofold: (i) individuating which auditory and visual cues are 
most involved in conveying the performer’s expressive intentions and (ii) test- 
ing the model developed by comparing its performance to spectators’ ratings 
of the same musical performances. 

In this experiment, expressive gesture was analysed with respect to its 
ability to convey the intensity of emotion to the audience. The study focused 
on communication through visual and auditory performance gestures of emo- 
tional intensity and the effect of it on spectators’ emotional engagement. 

The research hypotheses combine hypotheses from Laban’s Theory of 
Effort”? with hypotheses stemming from performance research” and research 
on the intensity of emotion and tension in music and dance:** 


21 More detailed description of such an experiment is available from Timmers et al. 
2006. 


22 Laban 1947, Laban/Lawrence 1963 
23 Palmer 1997; Timmers 2002 
24 Krumhansl 1996; Krumhansl/Schenck 1997 
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1. Emotional intensity is reflected in the degree of openness (release) or 
contraction (tension) of the torso of the performer. 

2. Emotional intensity is communicated by the main expressive means 
for a pianist: tempo and dynamics. 

3. Intensity increases and decreases with energy level (speed of move- 
ments, loudness, tempo). 

4. Intensity is related to the performer’s phrasing: it increases towards 
the end of the phrase and decreases at the phrase boundary with the 
introduction of new material. 


4.1 Method 


4.1.1 Musical performance 


A professional pianist was asked to perform an emotionally engaging piece 
of his choice at a concert that was organised for the experiment’s purpose. He 
performed the piece first without public in anormal manner and an exagger- 
ated manner and then with public in a normal, concert manner. Exaggerated 
meant with enhanced expressivity, which was, according to the pianist, con- 
sistent with the style of performance of the early 20th Century. 

He performed on a Yamaha Disklavier, which made it possible to regis- 
ter MIDI information of the performance. In addition, audio recordings were 
made, and video recordings from four sides (Fig. 2). The video recordings from 
the left were presented to the participants of the experiment. 

The pianist chose to perform Etude Op. 8 no. 11 by Alexander Skriabin, 
which is a slow and lyrical piece (Andante cantabile) in a late Romantic style 
that has a considerable amount of modulations. According to the pianist, the 
piece can be played with a lot of freedom. Theoretically, the piece has a simple 
ABA with coda structure (A A B A’ A” C, to be more precise), but the pianist 
interpreted the line of the music differently: the first main target of the music 
is a release of tension halfway through the B section. Everything preceding 
this target point is a preparation for this tension release. The A section is 
anyway preparatory; it leads towards the start of the B section, which is the 
real beginning of the piece. After this release of tension, the music builds up 
towards the dramatic return of the theme of the A section. This prepares for 
the second possible point of tension release halfway through the coda at a 
general pause. The release, however, is not continued, and the piece ends 
most sadly. 
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Fig. 2. Video recordings of the piano performances (right, top, left, and front 
views) 
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4.1.2 Participants 


12 people participated in the experiment; among them were four musi- 
cians. The participants varied greatly in musical experience. Some of them 
never had had music lessons and hardly listened to classical music, while 
others had basically performed classical music for their entire lives. 


4.1.3 Procedure 


The participants saw the performances on a computer screen and heard 
them on loudspeakers. They saw and heard the performances two times. 
During the first hearing, they indicated the phrase boundaries in the music 
by pressing the button of the joystick. During the second hearing, they indi- 
cated to what extent they were emotionally engaged with the music by mov- 
ing a MIDI-slider up and down. The order of the repeated performances was 
randomised over participants. The whole procedure was explained to them by 
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a written instruction and a practice trial. 


4.2 Analyses 


4.2.1 Auditory Performance Data 


The key velocity and onset times of notes were extracted from the MIDI 
files (layer 1). From this, the average key velocity for each quarter note was 
calculated as well as inter-onset intervals (IOI’s) between successive quarter 
notes (layer 2). The quarter note IOI is an accurate measure of local duration, 
while key velocity corresponds well to local loudness. These measures were 
interpreted as a direct expression of emotional intensity and as an expression 
of musical phrasing.” 


4.2.2 Visual Performance Data 


For the analysis of the movement of the pianist, we concentrated on the 
movement of the head, which shows both backward-forward movement (y-di- 
rection) and left-right movement (x-direction). The position of the head was 
measured, using the Lucas and Kanade feature-tracking algorithm?® that 
assigns and tracks a specified number (in our case 40) of randomly assigned 
moving points within a region (layer 1). Velocity and acceleration were cal- 
culated for each trajectory using the symmetric backward technique for the 
numeric derivative (layer 2). Average values of position and velocity among 
the forty trajectories were calculated for both the x and y component. In 
addition, the velocity values were integrated for the x and y movement to get 
a general measure of amount of movement over time. Redundancy in the 
number of points (i.e., forty points instead, for example, of just the barycentre 
of the blob) allowed us to get more robust and reliable values for velocity. A 
low-pass filter was applied to smooth the data obtained. Measures were sum- 
marised per quarter-note in order to be comparable to the other measures. 


4.2.3 Spectators’ ratings 


For each quarter note in the performance, the number of people who indi- 
cated a phrase boundary was calculated by summing the number of bound- 
ary indications per quarter note over participants. This sum per quarter note 
was expressed as a multiple of chance-level, where chance-level corresponded 
to an even distribution of the total of segment-indications over quarter notes. 
This segmentation measure will be abbreviated as SM. 

25 See43Results. SS 
26 Lucas/Kanade 1981 
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The indication of emotional engagement was measured at a sampling rate 
of 10 Hz using a MIDI-slider that had a range from O to 127. The average level 
of the MIDI-slider (emotion measure, abbreviation EM) per quarter note was 
calculated for each participant separately. 

An EyesWeb patch application was developed to store and process partici- 
pants’ data in real-time. 


4.3 Results 


4.3.1 Auditory Performance Data (layer 3) 


The resulting profiles 
of quarter note key veloc- 
ity and quarter note IOI 


„Ol ($) 


were highly similar for the 


note 


three performances: they 


quarter 


all started at a slow tempo 
and with soft dynamics, 


and had considerable cre- 
scendi and accelerandi in 
the A section, a diminu- 
endo and crescendo in the 


y_per_quarter_note 


B section accompanied 
by first a highly variable 


key_velocit 


tempo and thereafter an 


ope tt 1. 14 


accelerando, a fast and 


bar 


loud return of the A sec- 

Fig. 3. The duration per quarter note and 
the key velocity per quarter note as it varies 
throughout the Skriabin Etude. Separate plots 
Jor the three performances of the piece. Vertical 
of the theme, and a coda lines indicate section boundaries. Arrows are 
that fades away in dynam- explained in the text. 
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tion with limited variation 
in tempo and dynamics, 
a soft and slower repeat 


ics and tempo (Fig. 3). This 
global pattern is indicated by arrows at the bottom of Figure 3. 

In addition to this global pattern, the IOI profile shows the characteristic 
peaks of phrase-final lengthenings. It shows this at a fairly high density and 
large magnitude, except in the forte return of the A section (A’). The key veloc- 
ity profile shows drops in velocity at most phrase boundaries, but these are 
compensated by strong crescendi in most sections. 
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4.3.2 Visual Performance Data (layer 3) 


Left-right movement The position of the 


— pl 


head is plotted in Figure 
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x My an p3 4 for two dimensions: left- 
a |> right (upper panel) and 
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backward-forward (bot- 
V tom panel). The move- 
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ment of the head was 


especially pronounced 
250 4 — pl and especially consistent 
san I N p3 over performances in the 
A left-right direction (cor- 
150 ] N 
] N a relation between pl and 
100 4 Vo ey W p2 and between p2 and 
I arai 8 at A c p3 was 0.79; it was 0.83 
0 Sr 
Bn eee gan: Sa. Ye) op „2 ae between pl and p3). The 
Fig. 4. The position of the head plotted per backward-forward move- 
quarter note. Upper panel shows left-right posi- ment becomes more pro- 


tion (x) and bottom panel the backward-forward nounced for the later per- 
position (y). Separate plots for the three per- 
formances of the piece. Vertical lines indicate 
section boundaries. 


ELLE relatively fast in the mid- 


formances (p2 and p3). 
The periodic movement is 
dle parts of the piece (B and A”) and slower in the outer parts. This suggests 
an intensification towards the middle followed by a relaxation towards the 
end. 


4.3.3 Relation between performance data 


Correlations between performance measures were calculated to check the 
coherence between measures. Key velocity and IOI are negatively correlated 
(r = -0.51 on average). Velocity of head movement is positively correlated with 
key velocity (r = 0.45 on average) and negatively with IOI (r = -0.25 on average). 
The low correlation between values is partly due to the asynchrony between 
the periodicity of the measures. If peak values (maximum for key and move- 
ment velocity and minimum for IOI) per two bars are correlated, agreement 
between movement and sound measures becomes higher. Especially the two 
velocity measures turn out to be highly correlated (r = 0.79 on average for key 
and movement velocity, versus r = -0.38 on average for movement velocity 
and IOD. 
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All performance measures show a periodic increase and decrease. To 
check the relation between these periodicities and the musical structure, 
the location of mimima in key velocity, and maxima in IOI, x-position and 
y-position were compared to the location of phrase boundaries. Generally, the 
Skriabin Etude has a local structure of two-bar phrases. The forward and the 
left position of the performer were taken as start/end point for periodicity. 
IOI was most systematically related to the two-bar phrasing of the Skriabin 
piece, followed by key velocity. 55% of the phrase-boundaries were joined by 
a slowing-down in tempo. The other phrase boundaries were directly followed 
by a slowing down in tempo (a delay of 1 quarter note). For the key veloc- 
ity, 42% of the phrase-boundaries coincided with a minimum in key veloc- 
ity, 15% were anticipated by a minimum and 28% followed by a minimum. 
The period boundaries of the movement of the pianist hardly synchronised 
with the score-phrasing. The location of these boundaries varied greatly with 


respect to the two-bar score-phrasing. 


4.3.4 Relation between 
Performance and Listeners Data (layer 4) 


In this study, we had four hypotheses concerning the communication of 
intensity of emotion in musical performances. 

Hypothesis 1 predicts that intensity of emotion is positively correlated 
with backward-forward movement (y). This hypothesis is easily tested and 
contradicted: the correlation between listeners’ indication of intensity of emo- 
tion and backward-forward position is negative (r is -0.23, -0.50, -0.29 for pl, 
p2 and p3, respectively). It is also contradicted with respect to the other per- 
formance data: y-position is negatively correlated with velocity and positively 
correlated with IOI, which means that the performer moves forward in soft 
and slow passages and backwards in louder and faster passages. 

Hypothesis 2 predicts that tempo and dynamics cooperate to commu- 
nicate intensity of emotion. This is made problematic by the fairly low cor- 
relation between IOI and key velocity and by their different relation with the 
score-phrasing. Instead the performance data suggests a differentiation in 
function between the two expressive means, and tempo strongly communi- 
cates phrasing. 

Hypothesis 3 predicts high movement to correspond with intense dynam- 
ics and fast tempi. As we have seen in the previous section, dynamics and 
movement velocity agree more strongly than movement velocity and tempo. 
Especially the variation in velocity peaks corresponds. 

Hypothesis 4 relates phrasing to intensity of emotion. A clear phrase end- 
ing is predicted to coincide with a release in emotional intensity. 
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A series of multiple regression analyses was carried out. In the first analy- 
sis, quarter note IOI, key velocity, and movement velocity were used to predict 
EM. In the second analysis, the same variables were used to predict SM. In 
the third analysis, peak values per hyper-bar were used to predict average 
emotion measure per hyper-bar. All analyses were done directly and with a 
time-delay of one, two and three quarter notes of the performance data with 
respect to the listeners’ data. The best R? obtained will be reported. These 
were obtained with a delay of either zero or one for SM, and either two or 
three for the EM. 

SM was rather well-predicted by the model, given the Rs? of 0.34, 0.33, 
0.30 for pl, p2 and p3, respectively. From this model, IOI was the only sig- 
nificant variable. In other words, duration was a fairly good predictor of the 
variation in number of participants indicating a section-boundary. More par- 
ticipants indicated a phrase-boundary for longer durations. 

EM was well-predicted by the quarter note model, but even better by the 
second model that took the peak values per hyper-bar to predict the average 
EM per hyper-bar. The quarter note regression analysis had an R? of 0.45, 
0.68, 0.50 for pl, p2, and p3, respectively, while the hyper-bar peak value 
regression had an R? of 0.53, 0.82, and 0.56. Velocity was always the most 
significant variable, and was the only significant variable for the hyper-bar 
peak value regression. For the quarter note regression movement velocity also 
reached significance for p2 and p3, and IOI for p2. All Rs? are relatively high 
for p2, which suggests that the exaggerated expression of this performance 
increased communication. 

As a comparison, the analyses were repeated with x-position and y-posi- 
tion as independent movement variables instead of the more general move- 
ment velocity variable. The results did not improve or change from this altera- 
tion, instead x and y-position did not contribute significantly to any of the 
regressions. 

These results confirm a differentiation between expressive means: tempo 
primarily communicates segmentation, while dynamics communicates emo- 
tional intensity. Velocity of the movement is correlated with dynamics and 
may therefore also reflect emotional intensity, but the sounding parameters 
are the main communicative factors. 

The results are suggestive counter-evidence for hypothesis 4. The failure 
of tempo to explain variations in emotional intensity contradicts the theory 
that phrase-final lengthenings cause a release in emotional intensity. There 
is, however, another way in which phrasing and the dynamics of tension and 
release do relate, which is at a higher and more global level. Phrase-final 
lengthenings occur at a high rate and a local scale. At this local scale the rela- 
tion is weak. Major phrase boundaries that are indicated by a drop in tempo 
and dynamics are, however followed by a clear release in intensity. Moreover 
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the global variation of dynamics to which the variation in emotional intensity 
is so strongly related is the performer’s way of communication of the overall 
form: the first part is an introduction and builds up to the B section, which 
he considers as the real beginning of the piece. This beginning is again a 
preparation for the first target of the piece: the release of tension in the middle 
of the B section.” Hereafter, tension builds up towards the dramatic return 
of the theme, which leads via a repeat of the theme in contrasting dynamics 
to the second important target of the theme: the second possible release of 
tension at the general pause. After the general pause, the release is not given 
and all hope is lost. The piece ends most sadly. The pianist most skilfully 
expresses this interpretation in the patterning of dynamics.” The resulting 
phrasing is over the entire piece, with subdivisions at measures 22 and 36. 
The return of the theme is the culminating point of the piece, whereafter ten- 
sion can release. According to the pianist, this tension cannot, however, be 


fully resolved. 


4.4 Summary 


This study had two aims: (i) individuating which auditory and visual 
cues are most involved in conveying the performer’s expressive intentions 
and (ii) testing the model developed by comparing their performances with 
spectators’ rating of the same musical performances. The auditory and visual 
cues most involved in conveying the performer’s expressive intentions were 
hypothesised to be key velocity, IOI, movement velocity, and the openness or 
contraction of the performer’s posture. In addition, a relation between phras- 
ing and emotional tension-release was expected. 

The analyses of performance data suggested the opposite relation between 
emotional intensity and the performer’s posture. The pianist leaned forward 
for softer passages and backward for intensive passages. In addition it sug- 
gested a differentiation in expressive means, with tempo on one side, and key 
velocity and movement velocity on the other side. 

When relating the performer’s data to the listeners’ data, this differen- 
tiation in expressive means was confirmed. Tempo communicates phrase 
boundaries, while dynamics is highly predictive for the intensity of emotion 
felt. Emotional engagement correlated strongly with key velocity, which means 
that emotional engagement tended to increase with increase of dynamics and 
decrease at points of softer dynamics. This does not mean that soft passages 
were without emotional tension, but they were points of relative emotional 
relaxation. Hardly any evidence was found for movement cues influencing 
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27 See downward pointing arrows in Fig. 3. 


28 See arrows in the key velocity panel of Fig. 3. 
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listeners’ ratings. The sound seemed to be the primary focus of the partici- 
pants, and vision seemed subsidiary. The local phrase boundaries indicated 
by tempo did not lead to a release of emotional intensity. The modulation of 
dynamics over a larger time-span communicates the overall form of the piece 
and, at that level, intensity did increase and decrease within phrases. 


5. Applications of Multimodal Expressive Systems: 
the Case Study of Active Music Listening 


Music making and listening are a clear example of a human activity that 
is above all interactive and social. However, nowadays mediated music mak- 
ing and listening is usually still a passive, non-interactive, and non-context- 
sensitive experience. The current electronic technologies, with all their poten- 
tial for interactivity and communication, have not yet been able to support 
and promote this essential aspect of music making and listening. This can be 
considered a degradation of the traditional listening experience, in which the 
public can interact in many ways with performers to modify the expressive 
features of a piece. 

The need to recover such an active attitude with respect to music is 
emerging strongly, and novel paradigms of active experience will be devel- 
oped. By active experience and active listening we mean that listeners are 
enabled to interactively operate on music content, by modifying and molding 
it in real-time while listening. Active listening is the basic concept for a novel 
generation of interactive music systems, which are particularly addressed to 
a public of beginners, naive and inexperienced users, rather than to profes- 
sional musicians and composers. 

Active listening is also a major focus for the new EU-ICT Project SAME 
(Sound and Music for Everyone, Everyday, Everywhere, Every Way).?? SAME 
aims at: (i) defining and developing an innovative networked end-to-end 
research platform for novel mobile music applications, allowing new forms of 
participative, experience-centric, context-aware, social, shared, active listen- 
ing of music; (ii) investigating and implementing novel paradigms for natural, 
expressive /emotional multimodal interfaces, empowering the user to influ- 
ence, interact, mold, and shape the music content, by intervening actively 
and physically into the experience; and (iii) developing new mobile context- 
aware music applications, starting from the active listening paradigm, which 
will bring back the social and interactive aspects of music to our information 
technology age. 
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In the direction of defining novel active listening paradigms, we recently 
developed a system, the Orchestra Explorer,” allowing users to physically 
navigate inside a virtual orchestra, to actively explore the music piece the 
orchestra is playing, and to modify and mold in real-time the musical per- 
formance through expressive full-body movement and gesture. By walking 
and moving on the surface, the user discovers each single instrument and 
can operate through her expressive gestures on the musical piece which the 
instrument is playing. The interaction paradigm developed in the Orchestra 
Explorer is strongly based on the concept of navigation in a physical space 
where the orchestra instruments are placed. The Orchestra Explorer is 
intended for use by a single user. 

Our novel multimodal system for social active listening, Mappe per 
Affetti Erranti, starts from the Orchestra Explorer and the lessons learned in 
over one year of permanent 
installation of the Orchestra 
Explorer at our site at Casa 
Paganini, and several instal- 
lations of the Orchestra Ex- 
plorer at science exhibitions 
and public events. 

Mappe per Affetti Erranti 
extends and enhances the 
Orchestra Explorer in two 
major directions. On the one 


hand it reworks and extends 


the concept of navigation by Fig. 5. A group of users interacting with the 


introducing multiple levels: installation “Mappe per Affetti Erranti” at 
the auditorium of Casa Paganini. 
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from the navigation in a 
physical space populated by 
virtual objects or subjects (as it is in the Orchestra Explorer) up to the navi- 
gation in virtual affective, emotional spaces populated by different expressive 
performances of the same music piece. Users can navigate in such affec- 
tive spaces by their expressive movement and gesture. On the other hand, 
Mappe per Affetti Erranti is explicitly designed for use by multiple users, and 
encourages collaborative behaviour: only social collaboration allows a correct 
reconstruction of the music piece. In other words, while users explore the 
physical space, the (expressive) way in which they move and the degree of 
collaboration between them allow them to explore at the same time an affec- 


tive, emotional space. 
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The basic concept of Mappe per Affetti Erranti is the collaborative active 
listening of a music piece through the navigation of maps at multiple levels, 
from the physical level to the emotional level. 

At the physical level the space is divided in several areas. The voice of a 
polyphonic music piece is associated to each area. The presence of a user 
(even a single user) triggers the reproduction of the music piece. By explor- 
ing the space, the user walks through several areas and listens to the single 
voices separately. If the users stays in a single area, she listens to the voice 
associated to that area only. If the user does not move for a given time inter- 
val, the music fades out and turns off. 

The user can mold the voice she is listening to in several ways. At a low 
level, she can intervene on parameters such as loudness, density, amount of 
reverberation. For example, by opening her arms, the user can increase the 
density of the voice (she listens to two or more voices in unison). If she moves 
toward the back of the stage the amount of reverberation increases, whereas 
toward the front of the stage the voice becomes drier. 

At a higher level the user can intervene on the expressive features of the 
music performance. This is done through the navigation of an emotional, 
affective space. The system analyses the expressive intention which the user 
conveys with her expressive movement and gesture, and translates it in a 
position (or a trajectory) in an affective, emotional space. Like the physical 
space, such affective, emotional space is divided in several areas, each one 
corresponding to a different performance of the same voice with a different 
expressive intention. Several examples of such affective, emotional spaces are 
available in the literature, for example the spaces used in dimensional theo- 
ries of emotion?! or those especially developed for the analysis and synthesis 
of expressive music performance.’ 

Users can thus explore the musical piece in a twofold perspective: nav- 
igating the physical space they explore the polyphonic musical structure; 
navigating the affective, emotional space they explore music performance. A 
single user, however, can only listen to and intervene on a single voice at a 
time: she cannot listen to the whole polyphonic piece with all the voices. 

Only a group of users can fully experience Mappe per Affetti Erranti. In 
particular, the musical piece can be listened to in its whole polyphony only if 
a number of users at least equal to the number of voices is interacting with 
the installation. Moreover, since each user controls the performance of the 
voice associated to the area she occupies, the whole piece is performed with 
the same expressive intention only if all the users are moving with the same 
expressive intention. Thus, the more users move with different, conflicting 
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31 See for example Russel 1980; Tellegen et al. 1999. 
32 See for example Juslin 2000; Canazza et al. 2000; Vines et al. 2005. 
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expressive intentions, the more the musical output is incoherent and chaotic. 
But the more users move with similar expressive intentions and in a collabo- 
rative way, the more the musical output is coherent and the musical piece is 
listened to in one of its different expressive performances. 

Mappe per Affetti Erranti can therefore be experienced at several levels: 
by a single user who has a limited but still powerful set of possibilities of 
interaction, by a group of users who can fully experience the installation, and 
by multiple groups of users. In fact, each physical area can be occupied by a 
group of users. In this case each single group is analysed and each partici- 
pant in a group contributes towards intervening on the voice associated to the 
area the group is occupying. Therefore, at this level a collaborative behaviour 
is encouraged among the participants in each single group and among the 
groups participating in the installation. 

The possibility of observing a group or multiple groups of users during 
their interaction with Mappe per Affetti Erranti makes this installation an 
ideal test-bed for investigating and experimenting group dynamics and social 
network scenarios. 


6. Discussion 


The modelling of expressive gesture is being accorded growing importance 
from both research and industry communities, even if we can consider it as 
being in its infancy. The main outputs of our research are the definition of a 
unified multimodal conceptual framework for expressive gesture processing, 
the experimental results obtained from the two described experiments, anda 
collection of software modules for cue extraction and processing. The concep- 
tual framework proved to be useful and effective in two different scenarios, 
well represented by the two experiments described in the paper. 

In the first experiment, we focused on the communication of basic emo- 
tions from a dancer to the audience, while in the second experiment we 
focused on the mechanisms that possibly cause emotional engagement in 
the audience. 

The dance experiment can be considered as a first step and a starting 
point toward understanding the mechanisms of expressive gesture commu- 
nication in dance. A collection of cues that have some influence in such a 
communication process was individuated, measured, and studied. A first 
attempt of automatic classification of motion phases was carried out and 
some results were obtained (e.g., an average rate of correct classification 
which was not particularly high, but well above chance level). Some direc- 
tions for future research have also emerged. For example, other classification 
techniques could be employed and their performances compared with what 
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we obtained with decision trees. Some aspects in dance performance that 
were only marginally considered should be taken into account. In particular, 
aspects related to rhythm should be further investigated. Expressive cues 
such as impulsiveness and fluency should be further worked out. Moreover, 
perceptual experiments would be needed to empirically validate the expres- 
sive cues extracted. 

The music experiment can be considered as a first step towards the 
understanding of the relation between movement and sound parameters of a 
performance, their expressive forms and functions, and their communicative 
function for spectators. A next step should involve a larger variety of perform- 
ances and a larger collection of calculated cues, and cues should be fitted 
to the responses of individual spectators in order to get a deeper as well as 
broader understanding. 

Expressive multimodal systems open a novel range of applications. At the 
end of this chapter we focused on an application in the field of active music 


listening. 
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INTERACTIVE 
MEDIA 
PERFOR- 
MANCES: 


PAST, PRESENT, 
AND FUTURE 


S 


RESONAN 
PARADIGM 


Martina Leeker 


All movement causes oscillation, 
yet we lack the ears to hear it. 
Marin Mersenne 


The subjects of this paper are theater/performance formats of contem- 
porary dance as exemplary instances of the interaction between people and 
media. Thus, practices of dancing with computers and the generation of 
music and sound in so-called real-time are interrogated.' This entails the dis- 
cussion of situations where the impression occurs that dancers create “their” 
own music by their movements. 

I intend to consider contemporary practices of dancing with comput- 
ers in their relation to a highly problematic historical discourse. Central to 
this discourse is the paradigm of a resonance between human beings and 
media constituted around 1900, which emerges through technical media 
and within the history of knowledge these technical media are based on. A 
frame of reference for the Resonance Paradigm? is the encounter with elec- 
tricity in the course of electromagnetic research. In this shape, it is the basis 
of signal theory in physics as well as in the experimental human sciences, 
the emerging physiology. While many of the phenomena observed in elec- 
tromagnetic research could not be scientifically explained at the time, they 
were implemented in technology used by media such as the telegraph or 
cinematography.* This gap between knowledge and technology frames the 
discourse of an “Aether Physics” influenced by spiritism. Aether Physics pro- 
vides the foundation for the Resonance Paradigm, which - in turn — produces 
newly invented, overstimulated “trigger-bodies” (Schaltkörper) dancing on the 
stages of theaters. Crucial for a critical reading of the Resonance Paradigm 
is the masking and camouflage it provides for an epistemic shift induced by 
electromagnetic research: from a conduction model of electricity to induction 
and thus to a notion of signal processing beyond language and meaning. 
This shift is masked by various suppositions of similarities between human 
beings, devices, and media on the one hand and the superimposition of the 
non-human with spiritism on the other hand, thus opening a channel of com- 
munication between a para-physical world and humanity. 
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1 Good examples of contemporary practice are the works of blackhole-factory (2001). 


2 Resonance occurs, once at least two entities start to oscillate and once they are 
oscillating in the same frequency amplify each other’s vibration, which presupposes a 
similarity of the participating entities. 

3 About film as scanning and signal processing, see Siegert (2006a) and Herrmann 
(1996). 
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1. Resonances in Technical Media 
around 1900 and the Interaction with Computers 


Around 1900, both discourse networks? and dance come into the reach 
of technical media such as phonography, cinematography and telegraphy, 
which allow the notation of the physically real. All of a sudden, the acoustic 
and the visual are frequencies. Entities that had been simply non-recordable 
and in-visible, now take shape according to the properties of new technical 
media. Simultaneously with these “new” media and new devices, dancers 
appear whose claim for the practice of dancing rests in their lack of formal 
training, since this very lack best suits a new kind of dance. Forms of dance 
emerge which are not touched by language but are constituted by sheer 
physiology, by a transmission of stimuli from body to body. In this context, 
dance and music turn into continuous imprints on the physically real and, 
furthermore, to options for regulating and optimising hitherto uncontrollable, 
preverbal organicity. 


Ma(g)deleine Guipet: 
Automaton of Reflexes and a Physiological Aesthetics 


Madeleine Guipet may well be seen as the defining instance of a new 
relation between dance and music, created and practiced in the realm of 
electron physics and spiritism. She leads the way in the transposition of 
the organic into technical media®, into trigger-bodies®. Guipet belongs to a 
group of hysterics, who had become something of the great white hope for an 
intensification of physical presence and the heightening of physical abilities. 
The female hysterics are seen as over-stimulated and over-sensitive subjects 
whose pathological disposition makes them resonate with the whole of their 
environment. Somewhat drastically, Albert von Schrenck-Notzing, the physi- 
cian in charge of Mlle. Guipet’s therapy, calls her a “somnambulist autom- 
aton of reflexes.” From the 1870s on, the French psychiatrist Jean-Marie 
Charcot uses hypnosis, the newly established experimental science, to draw 
the female medium into a state of trance, thus switching off cognitive control. 
4 The term “discourse networks” has been used to translate Friedrich Kittlers 
notion of “Aufschreibesysteme” (Kittler 1985) apparently for reasons of resonance with 


Foucaultian terminology (Wellberry 1992, p. XI). A more literal translation would read 
“systems of writing down” or “notation systems” (note of the translator). 


5 The notion of “technical media” is not meant metaphorically here, but in a very 
specifically technical sense. Bodies turn into technical media by being defined and 
addressed by contemporary automatons of experimental physics and physiology. 

6 Georges Didi-Huberman (2004, p. 192) introduces this term in his discussions of 
“Repetitions, Rehearsals, Staging” (pp. 175-257) as elements of the photographic prac- 
tices in Charcot’s Salpétiére. 


7 von Schrenck-Notzing 1904, p. 117 
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In this way, hypnosis turns into a controlling function of the human power 
plant of hysteria, since it turns the hitherto uncontrollably twitching ladies, 
who had been interesting for their intensified stimulatedness and sensitivity, 
into “trigger-bodies” where an archive of gestures and facial expressions could 
be retrieved and made visible in a precise manner.’ If the human body had up 
to his point primarily been the site of language, of the subject and the soul, 
it now turns into a terrain of technically reproducible command circuits; a 
constitution that humans share with animals and telegraphs. 

Madeleine Guipet is put into a trance on stage;? she is exposed to piano 
music, which - in the eyes of her physicians and her audience around 1904 
- forces her to dance like a puppet on strings.!° The theory explaining this 
phenomenon claims that the sound of the instrument consists of frequencies, 
which induce oscillation in the body of the medium, thereby activating pro- 
grams of stimulation stored in the body as neurological circuits. Madeleine is 
celebrated as the perfect example of the sensitised body and by extension, of 
the human potential for retrieving and unlocking ever-increasing powers of 
perception and knowledge. 

In a Nietzschean vein, Madeleine Guipet could be framed as evidence for a 
physiological aesthetics, an aesthetics constituted beyond language by being 
a pure transference of stimuli from body to body.!! Symbolic bodies thus turn 
into technical media, into trigger-bodies, and this is the final frontier of dance 
as an art form, since it is not art that is at stake here any longer, but the 
intensification of stimuli and biopolitics.'? 


Epistemic Shifts: Electromagnetism and Spiritism 


Studying the (false) belief in a resonance between people and media, 
which also echoes McLuhan’s notion of “media as extension of man” in mid 
20% century media studies,' reveals the epistemic situation of the late 19" 
century. The Resonance Paradigm emerges at a time of revolutionary changes 
in both physics and media technologies, which in turn foster insecurities and 
shifts in philosophical and epistemological means and concepts. In my view, 
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8 Herrmann 2005, p. 191 

9 Bahr 1999 

10 Herrmann 2005, pp. 202-211 
11 Herrmann 2005, pp. 195-200 


12 Hans-Christian von Herrmann notes that this increase in potential has always 
been a point of entry for biopolitics. If phenomena which had not been measurable turn 
into subjects for notation, they may also be reproduced and turned into training devices. 
Evidence for this dynamics may be found in the research and practical methods devel- 
oped with the cinematograph and chronophotography by Frank Bunker Gilbreth in the 
context of ergonomics. 
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the Resonance Paradigm - fueled by spiritist insanity — bridges two episte- 
mologies. It serves the purpose of blunting the impact of a cultural revolution 
by reworking notions of the anthropological'* and by the “expulsion of spirit” 
(Austreibung des Geistes)". 

The substantial turn at issue here is based on electromagnetic research." 
Its initial guiding question was an explanation of electromagnetism in accord- 
ance with calculable and predictable laws of nature - such as the law of 
conservation of mass/matter — as they had been known up to this point. 
The discovery of electromagnetic sparks by Heinrich Hertz (1886-1888) raises 
issues that cannot be resolved for the time being, since the spark leaps over 
a gap without being conducted by any materially specified substance. This 
phenomenon raises not only the all-decisive philosophical issue if and how 
the universe and the world could still be observed and comprehended. It also 
raises the question, whether a world in which something belonging to nature 
can leap may still be explained by one, spiritual, and all-encompassing prin- 
ciple. If electrons may be waves as well as particles, then - thus the inconven- 
ient truth - the traditional laws of classical Newtonian physics could perhaps 
not explain them and neither could they be calculated by a mathematics!” 
believing in its reference to the world.'® 

This loss of direct access to the world occurs on several levels. Crucial for 
an understanding of dance and music around 1900 for instance, is Hermann 
von Helmholtz’ research about the “Sensations of Tone” in the 1870s. It dem- 
onstrates that it is not the world that is heard but the brain itself, so that the 
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14 Siegert 2006b 


15 See Friedrich Kittlers Austreibung des Geistes aus der Geisteswissenschaft (1980) 
where he laid the foundations for his media theory, namely the provocative argument of 
a mediatechnological apriori of anthropology and culture. The situation around 1900 is 
related to this notion, not in terms of media technology, however, but in the practices of 
camouflaging machines and as a discourse of its own. Major dramaturgical efforts are 
necessary, after all, to deliver human beings to the machine. 


16 Hagen 2005 


17 The role of mathematics within the turn to a theoretical physics of approximation 
and uncertainty relations initiated by electromagnetism can only be sketched here. At 
first, mathematics obstructs the inevitable paradigm shift. Although its crisis of foun- 
dation turns it towards a self-referential axiomatic model (Hilbert), mathematics also 
comes up with the new metaphysical concept of the inclusive calculability of the world. 
Suffice to mention, that Cantor presented in 1880 the Mannigfaltigkeitslehre (known 
today as set theory), which tries to calculate (aether’s) infinity and make it computable. 
Hilbert’s Axiomatic Mathematics, as well as the formal logic that rose from it and became 
the basis for computers, continue this tradition of abstraction and formalisation towards 
the ends of calculating the infinite and purely spiritual (and thus of the aether). For an 
introduction to the interplay of physics and mathematics in the context of spiritist ways 
of thinking, see Hagen (1995/96). 


18 According to Wolfgang Hagen, the discovery and exploration of electromagnetism 
was a turning point that turned thinking and knowledge away from the philosophy of 
nature and towards a world of approximations and uncertainty relations, where world 
and formal systems co-exist side by side. 
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human body, its nervous system pervaded by electrical circuits, constitutes 
perception.!? If we had different ears, brains and nerves, we would be hear- 
ing differently. In the wake of Helmholtz, the issue of perception turns from 
the transport and passage of impulses to their transmission and to induc- 
tion as an ensemble of triggering events and circuitry within the confines of 
previously stored programs. It is all about the control and the operation of 
information. 

The phenomenon emerging in the realm of physiology, namely the sepa- 
ration of energy from information, is implemented in the media technologies 
of telecommunication. The terrain where insecurities raised by the episte- 
mology of electromagnetism crystallise most tangibly is telegraphy, which 
emphatically articulates the bond between telecommunications technology 
and Aether Physics - the kind of physics that still believes in the existence of 
a spiritual and, at the same time, materially tangible soul of nature, despite 
emerging doubts. 

Furthermore, telegraphy demonstrates that in information technology 
neither ghosts nor energies, neither rays nor currents are being switched, but 
only information. And similar to electrons, information has no business with 
the anthropologically meaningful. If Helmholtz had already reduced the sense 
of tone to electrical impulses whose fate was bound to the facilities of the 
brain, the separation of energy and information finally creates an empire fun- 
damentally at odds with human facilities of comprehension, since meaning is 
nothing and signals are all. Any relation of signifier and signified is severed 
and communication may no longer be comprehended in terms of comprehen- 
sion but merely as transport of signals. 

This new disposition places human beings in an uncertain position 
between modalities of conduction and switching, between spirit and informa- 
tion — a site emerging from the differentiation of energy and signal, of matter 
and information in electrical processes. The potentially threatening effect of 
the electrically electronic media - that is the technical media - is an erosion 
of the anthropomorphous. They create a realm framed by analogue signals, 
which could still be human and thus comprehensible (not the least because 
they make him or her dance), and information, which human beings can- 
not comprehend any more. Human existence thus turns into a co-existence 
with self-organising machines and with White Noise, where meaning is either 
some kind of accident or a result of programming. 

It is this situation, where Aether Physics comes to the rescue since it 
establishes connections between media, physics, experimental sciences and 
spiritism, which bring incomprehensible media and their momentum of with- 


nn 
19 For the importance of resonance or “Mittönen” in Helmholtz’ psychophysiology of 
hearing on the basis of soundwaves and Fourier analysis as discrete formations with no 


orientation in meaning, see Volmar (2003). 


drawal from human comprehension into a horizon of explanation. While this 
horizon is in itself not rationally comprehensible, (since it operates with phe- 
nomena of the extra sensual), it is potent and seemingly salutary for the 
strain of epistemological shifts. Within the awkward philosophical, theologi- 
cal, and media-technological situation, Aether Physics may be seen as an 
attempt to provide access to phenomena of electromagnetism no longer or 
not yet explainable by laws and methods of classical physics. Yet at the same 
time, Aether Physics claims to have evidence of a “fourth dimension” consist- 
ing ofrays and fluids endowed with spiritual power, the aether indeed. Thus, 
William Crookes” proclaims: 


It seems to me that in these rays we may have a possible mode of transmit- 
ting intelligence, which, with a few reasonable postulates, may supply a 
key to much that is obscure in psychical research. Let it be assumed that 
these rays, or rays of even higher frequency, can pass into the brain and act 
on some nervous centre there. Let it be conceived that the brain contains 
a centre which uses these rays as the vocal cords use sound vibrations ... 
and sends them out, with the velocity of light, to impinge on the receiv- 
ing ganglion of another brain. In this way some, at least, of the phenom- 
ena of telepathy, and the transmission of intelligence from one sensitive to 
another through long distances, seem to come into the domain of law, and 
can be grasped.?! 


In favour of spirits and ghosts, which may neither be seen nor heard, 
the leaping sparks — as much as the electrons whose dynamics are as such 
invisible, yet leave traces of their operations on other, conducting media — are 
thus denied the status of being real by themselves. This resolves issues aris- 
ing from the new technical media’s seeming abilities to record the physically 
real itself - as opposed to the semiotic-hermeneutical systems of notation 
employed by writing and the notation of dance. Thus, the recordings of fre- 
quencies of voices, sound and movement in resonance with the human body 
are displaced into the Aether and explained as phenomena of spiritual and 
spectral transmissions - all implications of insanity and occult spiritualism 
included. 


Aether Physics in Performance: 
Occultism and Bodies as Technical Media 


The turn in media history and the history of knowledge from the physics 
of electricity and its epistemology to the era of communications technology 


A 
20 William Crookes was a physicist and chemist. Since he conducted experiments with 
vacuum tubes, he is widely credited for the discovery of the cathode ray tube. He also 
experimented with various spiritistic media and was a member of the Theosophic Society 
as well as president of the Society for Psychical Research. See Hagen (2005) and Siegert 
(20060). 


21 Crookes 1897, p. 338 
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and Quantum Physics denies feasibility to traditional concepts maintaining 
a unified comprehension of and access to the world. Within the confines of 
this situation, the dancers of hysteria may well be read as bridging a gap 
between two epistemologies — albeit fuelled by spiritism and insanity. Their 
cultural productivity derives from their ability to prevent a complete, concep- 
tual collapse of the human being by abolishing the anthropological altogether. 
The hysteric, musically-induced dancers may well be seen as hinges between 
energy and information. They are an attempt to conserve some of the Aether, 
before Axiomatic Mathematics, computers and Quantum Physics turn aside 
the electrified universe and the anthropological, or rather try to fuse both into 
the information machine. 

Bodies as technical media achieve this delay, yet also break the ground 
for the integration of human beings into informatics: by the mediation of 
resonance and embeddedness in a model of oscillation contextualised in 
the occult on one hand, and by opening a passage towards the discrete, the 
transformation of the anthropological into information, on the other hand. 
Bodies as technical media thus provide a training ground for the trespassing 
of the threshold and simultaneously liquidate discreteness. The technological 
epitome of this risky position between energy and information is the relay, the 
switch, separating energy and information.” Thus, bodies turn into relays or 
trigger-bodies. Yet inside of them, information also turns into oscillation, is 
triggered into resonances with the person and thus its ghostly apparition may 
still be humanly comprehensible and controllable. That this embodiment of 
media means to inscribe insanity into media, people and human-media, is 
the price to pay. 


2. Interaction and forced Immediacy: 
Digital Operations since 1960 


As we know from cybernetics,”* this scenario of camouflaging the his- 
tory of knowledge, of epistemic shifts and media in the resonance paradigm, 
continues well after the actual switch to discrete processes in the mid-1940s. 
Human beings turn into information processing machines and cybernet- 
ics rises to the status of an all-encompassing discourse of explanation for 
humans, animals, learning, society, education, economy and culture.** All 
these entities are similar in respect to the engineered control of self-organ- 


22 Paradigmatic of this function are the relays at telegraph stations. By installing a 
relay on the side of the receiver, one does not have to send energy any more, but only an 
impulse, which hits the relay, moves it to switch a battery and thus close an electrical 
circuit for the transmission of signals. 

23 Pias 2003 


24 Pias 2004 


297 


ised processes following the principles of a formal logic of classification and 
address-management by means of feedback. In order to explore the archi- 
tecture of computers but also to construe interplay between bodies, percep- 
tion and machines, many performances since the 1960s have turned to the 
Resonance Paradigm. Strategies and approaches that attempt to create reso- 
nance by means of camouflage are contemporary additional elements, while 
historic arguments referring to para-psychological events have ceased.” 
Instead, the technological structure of computers is displaced” by connecting 
electronic processes via imitation to the imagery and epistemology of hysteric 
media and dancers. This camouflage of the symbolic, universal machine with 
the aid of electronics has materialised since the 1940s in developments like 
interfaces and analogue/digital conversions.’ In addition, there has been a 
notable stress on performativity since the 1980s and more emphatically in the 
1990s. Non-permanence, traces, intermedial transmission and transforma- 
tion have gained importance, and to some extent even turned into ontologies 
of the technical materiality of computers as well as the anthropological. May 
we surmise then, that performativity as mediality and mediality as performa- 
tivity have taken the place of spiritist and occult discourse??® 


Echoes: The Resonance Paradigm 
and Contemporary Interactive Performances 


As mentioned earlier, contemporary dance performances with comput- 
ers evoke the impression of dancers creating their own music by means of 
their movement. While these practices seem clearly opposed to the spasmodic 
electro-hysteric dance circuitry of the 19% century, they may also be seen as 
modifications of hysteric dance for the age of the computer. Thus, the dancer 
is no longer “wired” to the music like Mlle. Guipet, rather it may seem that 
she or he is in control of the computer. In as much as the computer seems to 


25 If spiritist hermeneutics have been debunked for good, it, however, remains a sub- 
ject of discussion. Areas of research like biofeedback, where the circuitry of the brain is 
to be made visible, are somewhat suspicious. This practice would have to be questioned 
for its genesis from theosophical concepts like those of Leadbeater, who assumed differ- 
ent auras of human beings, which materialised in energies and colours. 


26 Since the inauguration of cybernetics, this camouflage has lived off its own impos- 
sibility, since the analogue data of continuous organic processes cannot be transferred 
to modes of discrete scanning without considerable friction. Faster computing and scan- 
ning are needed, as well as a deception of perception by means of dramaturgies control- 
ling attention and an aesthetics of imitating the electronic. See Hagen (2002a). 


27 For a history of the ambivalence of analogue/digital conversion, see Schröter 
(2004). 


28 For a discussion of the introduction of performativity as status of media into media 
and particularly computer studies via concepts like trace, process, transformation and 
contingency, see Kramer (1998, 2001). For a critical discussion of performativity, see 
Winkler (2004). 
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transform human movement into the dance preceding its acoustic coding, the 
impression occurs that the difference between notation and performance as 
well as the difference between dance and music have finally been overcome, 
since performance is writing itself down. Dance thus would return to its 
“essential being” - an ephemeral occurrence in time, which seems to embody 
processuality and transformation as anthropological ontology. Yet behind this 
discursive superimposition lures the fact that human beings and computers 
are fused to the same feedback loop. 
Performances using so-called image / Clings i sleet seele 
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The Resonance Paradigm, based in spiritism and Aether Physics as it is, 
can hardly be articulated more blatantly. Assisted by McLuhan’s notion of 
media as externalisations and extensions of human beings, these contem- 
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29 Leeker 2005 
30 See <http:/ /www.cycling74.com/products/maxmsp>. 
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porary practices pass on the insanity of those dancers in hysteria who con- 
nected to machines around 1900.°! 


Technology Suppressed: 
On the Difference between Recursion and Mapping 


A closer look at the technologies involved however shows that the cod- 
ing of human motion according to statistical data is thoroughly suppressed 
in this setting. Suppressed and made invisible is, for instance, that it is the 
recording of movement with digital videocameras that generates this “music 
in motion”. Video images are read into computers as an analysis of differ- 
ences to be mapped on (pre-)programmed sets of data, which then are turned 
into outputs of acoustic phenomena. Just like the electrical-electronic experi- 
ments in motion research around 1900, this contemporary practice has very 
little to do with motion. Instead, we face a quite thorough transformation of 
motion by means of its discrete formalisation. This is not an inscription of 
motion but the generation of signals being fed into circuits and interfaced 
with an analogue output. And computers are operating independently from 
human beings according to protocols of calculation and address-management 
based on a coding that is self-referential. Computers may - in other words — 
receive signals and process them, yet they do not compute human beings and 
the world but - in mercilessly flawless recursions — only themselves. 

It is thus not motion captured in sound or codings, but motion disas- 
sembled into discrete parameters, which are transcoded to fit the unity ofa 
dataset consisting of zero and one. This data, relayed to units of sound, writ- 
ing, image, constitutes the output. Mapping is a process quite different from 
recursion within the autonomy of coding, as it takes place in computers on 
the level of symbolic calculation. Mapping is trans-coding and thus represen- 
tation or, differently put, it already has and always will happen in the realm 
of representation. 


Imitation and Interaction: 
Electrohysteria and the Camouflage of Technology 


The camouflages, suppressions and deceptions practised in these per- 
formances are strategies of reconfiguring computers as something they have 
technically never been. I would argue this reconfiguration of technology as the 
prerequisite for the association of humans and machines with the Resonance 
Paradigm around 1900. 

To imply that computers and discrete codings are in any way connected to 
human beings and motion is clearly the product of a discursive inscription, 
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which may draw on the still resonant insanity of resonance between man 
and media around 1900. The implied recursion®? turns out to be camouflage, 
coverage of technology, which in turn allows the discursive production of 
computers as media by means of aesthetic-dramaturgical practices. The two 
strategies used in this context are (1) interaction and (2) imitation. 

Covering technology creates the (1) aesthetic impression of interaction 
and the impression of flow is crucial in this context. More than anything it 
is flow, the staged immediacy of a relation between dance and music, that 
would turn computers towards human beings, by developing and training 
strategies which transcribe discrete operations into continuous phenomena 
and perceptions. This kind of imaginary interaction with computers not only 
blocks its intrinsically time-critical dimension, but also purges constituent 
elements of digital encoding such as the inclusion of defects from the sce- 
nario of signal transport. 

This configuration of interaction is finally staged as a resonance between 
performers and computers by summoning iconographies whose power derives 
from their traditional embedding in cultural memory. This is achieved by 
a simple strategy of technical and aesthetic imitation, since the aesthetic 
design, the output of images and sound imitates (2) the phenomena inscribed 
or made visible by electro-electronic devices: waveforms, frequencies, dis- 
torted sounds, electro-acoustic tonalities, and multiplications of images. 
This is to say, that these manipulations refer to technologies belonging to the 
realm of the electro-electronic; they may be generated based on the deflection 
of electrons and sound frequencies. The analogue surfaces of computers, like 
screens or - as far as the conversion of analogue data is concerned - the MIDI 
converter, are used to imitate these manifestations of the electrico-electronic. 
Performance with computers thus seems to have made little progress since 
Nam June Paik’s practice of an “aesthetics of deflected electrons”°®. Needless 
to say, this aesthetics of the electron obscures its sources, operates beneath 
the surfaces, under the cover of camouflages, and technically speaking is 
based on discrete coding. It does entail, however, a cultural, electro-electronic 
re-programming of computers, whose technological sedimentation and imple- 
mentation may even be observed in devices like MIDI interfaces, sensor inter- 
faces and software environments like Max/MSP/Jitter. Manipulation of sig- 
nal chains and circuits themselves, however, does not occur at all, or remains 
the exception. 


Daca 
32 More precise in terms of technology, we must speak of re-coding. If there were recur- 
sion, motion would write itself acoustically without mediation. Yet this only happens, if 
motion is already available as relay-able, discrete symbolic coding. Motion itself, how- 
ever, cannot discretely be addressed “directly”. 
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Interaction and the imitation of the electro-electronic superimposes the 
inputs of human beings and machines and thus computers are generated as 
a continuation of the kind of electro-electronic epistemology of around 1900, 
described earlier. Where computers are used and thus a technological trans- 
formation of flowing energy in digital codings and rhythms of information 
occurs, these very structures and events are covered. At the same time, the 
conversion of analogue data into discrete data that separates human beings 
from machines is buried under this cover.” Call it a tactical misapprehension 
of computers’ discrete “nature”. 


Interactive Performance with 
Computers, Electro-hysteria and Cybernetics 


Interactive performances with computers since the 1960s% have thus 
managed to (re)create the computer from nothing but the epistemology of the 
electro-physiological experimental settings around 1900. This in turn brings 
forth cybernetic computing whose “merit” it has been to induce an under- 
standing of human beings as information processing machines and to have 
liquidated the computer as a symbolic machine. 

The crucial aspect of this connection is the inclusion of the human organ- 
ism in the image of a system controlled by electrical currents and circuits that 
designs this organism as a control device for integration in cybernetic feedback 
loops.*° The electro-hysteric-cybernetic analogy of human beings and comput- 
ers created by performances are thus of particular productivity. They merge 
both entities to one system, which is to say, that human beings are now part 
of an autonomous feedback loop of discrete operations — a prerequisite for 
these two systems to form an interactive relationship at all. Performances thus 
realise the integrated cybernetic feedback loop for human beings and universal 
machines, although the latter operate based on invariant chains of symbols. 


3. Media Technologies: 
Towards a History of their Dramaturgy 


The history of philosophy and the history of knowledge prepared an epis- 
temological field, which allowed and allows devices, machines and cultural 
techniques to turn into media. Media emerged and emerge from discursive 
and cultural “programming”, which may crystallise in technologies, as 
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35 The connection of computers and performance started with the experiments of the 
artists group EAT at New York during the 1960s. See Büscher (1998). 
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demonstrated in the instance of the MIDI interface. Performative techniques, 
dramaturgies, perhaps the instance of play as such, perception and camou- 
flage are central to this process. 

For Media Studies, this may well mean that they must always deal with 
the performative and that they have to do so in very specific and precise 
terms. After all, the instances discussed in this paper show that theatrical 
practices like camouflage, deception, empowerment, mimicry and imitation 
have been practices of immense power. They managed to reconfigure the com- 
puter against its own technological history and thus bestowed it with the 
potential to create worlds accessible for human beings. 

For Theatre and Cultural Studies, these findings suggest approaches that 
seek proximity to the analysis of technical materialities, since this may well 
be the only way to retrieve and decipher the discourses and the cultural 
programming of media technologies - including those by one’s own scholarly 
discipline. 

There is little doubt that the interaction with computers in performances 
is a valid field of research in this context, since it refers to both foundations 
and training within a historical account of the dramaturgy of media technolo- 
gies. The Resonance Paradigm is a cornerstone of this history, not merely 
because it continues to resonate in the interaction with computers up to this 
very day, but also as a cultural technique generating media as well as modes 
of using them. 


Translated by Michael Barchet 
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1. Introduction 


The term “media art” will be used here for artistic activity which either 
uses or schematises digital technology interactively. Media art begins where 
the traditional film genre ends: with interactive and process-related digital 
narratives. The cinema builds up a relationship between audience and pres- 
entation, and the viewer is promised a collective experience. By contrast, 
interactive storytelling offers the observer an individually tailored encounter, 
where the YOU_ser! is navigating the story. Based on rules of staging, media 
art conjoins with other cultural forms of expression such as performance, 
dance, theatre, film, architecture, sound, design or fashion. Media art gets 
its poetic strength from the new possibilities of interacting with its audience 
and from the cross-over between cognitive science? and arts. In the last two 
decades, the authors have carried out media art research in institutes like 
Art+Com? in Berlin, the Academy of Media Arts in Cologne and currently the 
Fraunhofer Institute IAIS near Bonn. 

Up until today, the use and maintenance of media art has been tech- 
nically challenging. Hardware and software are changing constantly, and 
the works of art would have been adapted for every new ICT* generation. 
Therefore many works of media art from the 80s and 90s can no longer be 
seen. If not restored by the artist, the works are unlikely to be restored by a 
museum or a gallery, for financial reasons or because of lack of competence. 
Nevertheless, no other art form is so close to our present day electronic world, 
and it therefore deserves greater attention. Interactivity, narrativity and digital 
scenographic audiovisual production are important topics of new media art. 
In contrast to cinema, interactive media art orchestrates the senses e.g. by 
staging tactile elements — even virtual ones - on a virtual stage next to visuals 
and acoustics. Digital storage and statistical procedures make visible what 
would otherwise remain invisible. Participation of the spectator in process- 
ing interactive media art creates awareness and new knowledge (Erkenntnis). 
Our thesis is that interactivity supports performative communication and the 
creation of performative presence. 


MLA 
1 For interactive media art at ZKM exhibition “YOU_ser” see <http://on1.zkm.de/ 
zkm/stories/storyReader$5591>. 


2 Cognitive science is defined as the study of mind or of intelligence. The interdis- 
ciplinary study draws on relevant fields including psychology, psychiatry, philosophy, 
neuroscience, linguistics, anthropology, computer science, and biology. The term was 
coined by Christopher Longuet-Higgins in 1973 concerning the then-current state of 
Artificial Intelligence research. See <http: / /en.wikipedia.org/wiki/Cognitive_science>. 


3 In 1988, a group of architects, artists, designers, scientists and technicians, 
amongst them Monika Fleischmann, Wolfgang Strauss, under the direction of architect 
Edouard Bannwart, co-founded the collective Art+Com e.V., researching and developing 
information design and communication technology. 


4 ICT- Information and Communication Technologies 
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2. Interactive Media Art 
as Performative Communication 


In the 1980s and 90s, media artists experimented between science and 
art with the aesthetic potential of process-related image genres. Technological 
institutes, namely in Germany, the US and Japan, invited artists as research- 
ers. The art historian Oliver Grau (2004) writes: “Internationally renowned 
exponents such as Charlotte Davies, Christa Sommerer and Laurent 
Mignonneau, Monika Fleischmann and Wolfgang Strauss, Jeffrey Shaw or 
Victoria Vesna, work, as a rule, as scientists in research laboratories and, for 
example, develop new interfaces, interaction models and code innovations. 
So they set anew the technical limits according to their aesthetic aims and 
critical messages.” Experimentation stands at the forefront of the new telepre- 
sence. Via broadband, the Virtual Environment can be accessed worldwide. 

In autumn 1991, the authors sent an early version of the Virtual Reality® 
installation “Home of the Brain” over the ISDN data lines from Art+Com in 
Berlin to the exhibition space in Geneva. Visitors in Geneva moved with a 
data glove through a virtual Potsdamer Platz and the virtual New National 
Gallery. “Home of the Brain”, a virtual exhibition and communication envi- 
ronment, translated the antique Stoa concept of a public place for meeting 
and discussion into the virtual space. The visitor navigates with a data glove 
through virtual rooms that are made visible with data glasses. Hand move- 
ments activate the citations of four media thinkers who play an important role 
in the theoretical formation of media culture. They are represented alongside 
their theoretical concepts by individual thought buildings, literally speaking 
houses dedicated to Joseph Weizenbaum, Marvin Minsky, Paul Virilio and 
Vilém Flusser. At the end of the 1980s, the work was designed to reflect the 
current media discourse. 

“Home of the Brain” is staged as an encounter in areas of thought - a 
Virtual Denkraum. The visitor is part of the staging and becomes located in 
the minds of others. The current discourse is reflected through the medium 
itself. The virtual space is interwoven with light, shade, colour, texture, words 
and sound, and all together with movement. The art historian Oliver Grau 
describes the Virtual Reality installation as media theory put into practice, 
and a new mnemonic theatre that anticipated the form of communication 
with networks: “Home of the Brain’ therefore emerges as early as 1991 as 
an early appearance of the epistemic innovation telepresence. As a conse- 


Decca 
5 With the term “virtual reality”, communications technology has offered equipment 
and concepts which allow an entry into the virtual environment. Data gloves and data 
glasses bring the observer into the setting, as the authors show with “Home of the 
Brain”. See <http: / /www.medienkunstnetz.de/works/home-of-the-brain/>. 


6 Fleischmann / Strauss (Art+Com) 1990-1992 
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quence, the reception of the art work in this way 
loses its local fixation, the observer does not go to 
the work, to the panel, panorama, cinema film etc., 
however, the work does not come exclusively to the 
observer.”” 


3. Origins of Virtual Reality 


When we started our research in 1988, there 
was a virtual void. We were thinking about compu- 
tational models. We were looking for the underlying 
structure, for navigation and orientation in virtual 
space. There was a need for new paradigms of space 
and interactivity. How should we deal with telepre- 
sence? How should we organise information? How 
should we link information and interaction with vir- 
tual objects? We studied our material and looked 
for adequate methods. It was “like studying celluloid 
instead of cinema, paper instead of novels, cathode 
ray tubes instead of television, hardware instead of 
software.”® And we concentrated on the interface 
to explore the role of the senses in mixed reality 
space. 

Virtual Reality evolved from mechanical simula- 
tors for the training of combat pilots in the Second 
World War and computer graphics research in the 
early 60s. “Virtual Art”, based on Virtual Reality 
(VR), the making of interactive environments, had 
nearly no resources in the traditional art world to 
have recourse to. In “The Ultimate Display”, compu- 
ter scientist Ivan Sutherland (1965) published the 
theoretical foundations for VR: “The ultimate display 
would, of course, be a room within which the com- 
puter can control the existence of matter. A chair 
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7 Grau 2004 
8 Bates 1991 
9 For the history of Virtual Art see Grau 2003. 
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Fig. 1. Home of the Brain - VR Installation (1991) 
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displayed in such a room would be good enough to sit in. Handcuffs dis- 
played in such a room would be confining, and a bullet displayed in such a 
room would be fatal. With appropriate programming, such a display could 
literally be the Wonderland into which Alice walked.” There was the idea of 
a computer-based interactive fantasy system to ‘go anywhere and do any- 
thing’. With goggles and gloves the interface hardware and software problems 
were solved by Scott Fisher in 1986 at NASA Ames Research and Frederick 
Brooks in 1988 at the University of California.!° In our research we used VR 
as a medium to express an idea, a vision of future communication or about 
the future city after the Berlin Wall had fallen. The authors’ interactive table 
installation “Berlin, Cyber City”!! (1989-90) was the origin of the tabletop 
interface “Responsive Workbench”!? (1993-94). 


4. Space: From Performance Space 
to Performative Presence 


Where did inspiration come from? At first one described only such 3D 
real-time simulations as Virtual Reality, but then in the 1990s the inter- 
net itself was linked to William Gibson’s term ‘cyberspace’ (from cybernet- 
ics and space).'’ With ‘Johnny Mnemonic” from the short story collection 
“Cyberspace”, and with his novel “Neuromancer” from 1984, Gibson revo- 
lutionised the way people look at technology. But inspiration came not only 
from technology research, computer graphics and science fiction. Film, thea- 
tre, music, literature and pop culture also had an influence on the develop- 
ment of interactive media, virtual art and the creation of transformative spac- 
es.!* Virtual Reality is the fusion of real-time, space, interface, haptics and 
movement with image and sound in conjunction with mathematical thinking. 
In our artistic work we are interested in the perception and reflexion of the 
current situation of communication technologies. Since 1988 we have been 
working on cultural interfaces to link real and virtual space. We initiated our 


10 For an “American” Timeline of Virtual Reality see <http://chrishutchison.org/atti- 
caschool/vr /Issues/page3 1 /page31.html>. 

11 “Berlin, Cyber City” was invented by the authors with a team at Art+Com as an 
interactive table installation with the virtual reconstruction of the city of Berlin. See 
<http: / /netzspannung.org/ database /cyber-city/en>. 

12 The prototype of the “Responsive Workbench” was invented by the authors under the 
direction of Wolfgang Krueger and with the team at GMD - the German National Research 
Centre for IT. See <http: / /netzspannung.org/database/responsive-workbench/en>. 

13 For the term “Cyberspace” see on Wikipedia <http://en.wikipedia.org/wiki/ 
Cyberspace> and ARTE Portrait 2005: Dream - William Gibson. See <http://www. 
arte-tv.com/de/kunst-musik/tracks/Diese-Woche/20050203/804294.html> (last 
access: June 2008). 


14 Packer/Jordan 2002 
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own Labs’? to study and produce the interlinking of art, technology and sci- 
ence. In transdisciplinary teams we find the patterns that make a difference 
by using the epistemologies of each discipline to drive inquiry. We take theo- 
ries and methods which exist independently of several disciplines and apply 
them to organise and understand different areas for the purpose of achieving 
new insights. 

The VR installation “Home of the Brain” relates to concepts of the avant- 
garde theatre, such as Samuel Beckett’s “Quadrat”, where people react to one 
another and build up relationships by walking on predefined paths. We were 
inspired by the late Klaus Michael Grueber, the wanderer among the stage 
directors, who used unusual public spaces to appeal to the audience. The 
theatrical performance installation “Rudi” (1979) dealt with forms of remem- 
bering. The setting, near the — at that time still existing — Berlin Wall, in 
the prestigious Hotel Esplanade'‘, became a stage and exhibition space for 
an outstanding performance. It involved a tour through the house that sets 
thoughts into motion, although hardly anything in the space moves. An actor 
sits in front of a fireside and read out Bernard von Brentano’s 1934 novel 
“Rudi” in a monotonous voice and with many breaks - loudly. Parts of his 
reading were transmitted into the other rooms of the hotel through loud- 
speakers. The audience moved around as ifin an exhibition in order to incor- 
porate the idea of spatially performed information related to the voice, move- 
ment and stage. There was no linear narrative structure to follow. It was not 
clear: Is this reality, theatre or a museum? The borders of real and fictional 
space blurred and became one (mixed) reality. The audience was part of the 
concept of an aesthetic experience. It impressed and unsettled the people of 
Berlin in the ruins near Potsdamer Platz. The merging of theatre and museum 
in “Rudi” transferred the visitors into a situation of in-betweenness.'” The VR 
environments “Home of the Brain” or the subsequent “Murmuring Fields”!® 
refer to fundamentally different notions of space: the performative presence 
of becoming in the virtual space and the physical presence in the space of 
performance.'® During the press conference where “Home of the Brain” was 
presented with the Golden Nica of the Prix Ars Electronica in 1992 in Linz a 
visitor explored the VR environment. He moved his whole body as if he were 


0 zz 


15 First Art+Com, later the Fraunhofer MARS Exploratory Media Lab. See <http:// 
netzspannung.org/about/mars/projects/en>. 


16 The famous Hotel was damaged in the Second World War, but still became the scene 
of famous films. Today, the preserved parts of Hotel Esplanade can be visited in the 
Sony-Center at the Potsdamer Platz, where they were transferred in 1996 by means of 
complicated technology. See <http:/ /de.wikipedia.org/wiki/Hotel_Esplanade_(Berlin)>. 


17 Fischer-Lichte 2008 
18 Fleischmann/Strauss 1997-99 
19 Gemeinböck 2004 
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swimming around the virtual objects. He represented an in-betweenness of 
being in both spaces - the real and the virtual. 


5. Time: Film as Seismograph 
of Nonlinear Image and Sound Concepts 


Experimenting with Virtual Reality in the 80s and 90s, artists and sci- 
entists felt still influenced by the Apollo space-flight program undertaken 
by NASA during the years 1961 - 1975. They were affected by films such as 
“2001: A Space Odyssey” or “Powers of Ten” presuming the non-linearity and 
the endlessness of space that was first experienced with Virtual Reality tech- 
nologies — but then interactive and in realtime. “2001: A Space Odyssey” by 
Stanley Kubrick? is a science fiction epic and was flavour of the moment in 
1968: America was in space fever. The film is not only a future vision about 
contact with aliens and the endlessness of space, but also about the questions 
of life that are often interpreted by artists and philosophers: “Where Do We 
Come From? What Are We? Where Are We Going?”?! In an interview Kubrick 
said that he “tried to create a visual experience, one that bypasses verbalized 
pigeonholing and directly penetrates the subconscious with an emotional and 
philosophic content.””? Instead of using much dialogue,”* Kubrick achieved 
his goal by using music and sound atmospheres essentially by Gyorgy Ligeti, 
Richard Strauss and Johann Strauß. The film begins with a black image and 
Ligeti’s “Atmosphéres” (1961). In this piece melody and rhythm are blurred 
beyond recognition through the creation of sound complexes. It concentrates 
on the texture of the sound. The work is, as Ligeti said, “just a floating, fluc- 
tuating sound, (....) You hear a kind of impenetrable texture, something like a 
very densely woven cobweb.” Ligeti’s works create the suitably weird sound 
effects for the more “far out” trips of the film.” A feeling of hallucination was 
created, only with sound and images.” Critics characterise the film as being 


20 Interview with Stanley Kubrick “2001: The space odyssey” explaining a treat- 
ment of the film can be found on the internet by New Media Giants. See <http:/ /www. 
kubrick2001.com/> (last access: June2008). 

21 This was the title of Paul Gauguin’s philosophical painting from 1897/98. 

22 Stanley Kubrick by Eric Norden. Playboy Magazine. September 1968 

23 There are only 43 min. in this film of 143 min. length. 

24 For Program Notes of San Francisco Symphony see Gyorgy Sandor Ligeti, San 
Francisco Polyphony. See <http:/ /www.sfsymphony.org/music/ProgramNotes.aspx?id 
=28950>. 

25 György Sandor Ligeti, San Francisco Polyphony. See <http:/ /www.sfsymphony.org/ 
music /ProgramNotes.aspx?id=28950>. 

26 Kubrick used more than 33 min. of Ligeti’s “cluster sound”. Though he did it with- 
out the composer’s knowledge or permission, the film created a constantly growing inter- 
national interest in Ligeti’s music, not only in the classical but also in the popular 
world. 
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“hypnotically entertaining, and it is funny without once being gaggy, but it 
is also rather harrowing.”” The film’s well-known fanfare and title music, 
“Also Sprach Zarathustra, Op. 30” is a tone poem by Richard Strauss, com- 
posed in 1896 and inspired by Friedrich Nietzsche’s book of the same name 
from 1883/1885. The title sequence begins after the black image with the 
Earth rising over the Moon, while the Sun rises over the Earth. “Thus Spoke 
Zarathustra” was used in similar situations such as the TV coverage of the 
Apollo Moon missions and landings in the late 1960s and 1970s. For the 
space scenes Kubrick filled the vacuum of space paradoxically with “The Blue 
Danube”, the famous Waltz by Johann Strauss Jr.: “it certain[ly] suggests the 
dance of space craft under the slow inexorable influence of Newtonian gravity 
and mechanics. The Space Station pirouettes, while inside a member of the 
cabin crew demonstrates walking under zero gravity conditions while objects 
like pens float off.” Although the film does not adhere to the audience’s 
usual expectations — there is no action, no plot and no resolution — “2001” 
became one of the most successful films ever and a classic of cinema history. 
Alongside the lack of a plot, the characteristic of boundless und bottomless 
floating in space is similar to the sense of space in Virtual Reality, and the 
structure of Ligeti’s music is reminiscent of interwoven algorithms. 

Another example of seismographic, self-proclaiming new image concepts 
is the nine-minute film structured as a documentary, “Powers of Ten” by 
Charles and Ray Eames, which likewise arose in 1968 within the context of 
space research. It demonstrates zooming and scaling — further design prin- 
ciples of interactive media. The film takes the audience on a journey, which 
begins with a picnic on the seashore in Chicago and leads to the edge of the 
universe. Every ten seconds we can see the starting point of the journey 
from a distance ten times further away from the earth, which then can only 
be seen as a point of light between many others. On the return journey, the 
view is enlarged, with breathtaking speed, tenfold every ten seconds. The 
camera shows the sleeping picnicker, then reduced to the view of his hand, 
then into his hand, and ends up on the inside of the cells of one of his DNA 
molecules. Charles and Ray Eames, with this film, give an idea of the rela- 
tive size of objects in the universe. They show how a single idea can reflect a 
universe of thoughts. Questions of measurement shape our understanding 
of the world. Knowledge of measurement and scale — as shown here — change 
the perspective of all things. Both films from 1968 are very much supported 
by atmospheric sound. In both, the earth is seen from space and here it was 
seen for the first time as a vulnerable global home worthy of protection. This 
perspective gave us a feeling for our place in the universe. 


zz 


27 Gilliatt 1970 
28 “2001: A Space Odyssey - Original Soundtrack” online available <http:/ /www. 


mfiles.co.uk/reviews /2001-a-space-odyssey.htm> 


6. Locative Media for presenting the World 


Locative Media is a new form of land art, where artists reflect e.g. on sur- 
veillance as a new form of presence. Networked Nature, or Locative Media, 
has a history that predates that first satellite launch in 1978. Bleecker and 
Knowlton (2006) write about the origins of GPS-Enabled Locative Media: “For 
instance, the ‘Earthworks’ group exhibition in October of 1968 in New York 
may count as a canonical point in the history of such geography and land 
form inspired art works. There is a distinction to be made of motivation as 
well as technique, which is what we mean to draw out by demarcating pre- 
satellite from satellite-enabled locative media. (...) It is painfully ironic that, in 
a time when public funding for art in the U.S. has evaporated, locative media 
artists are able to “piggy-back” on the U.S. Department of Defense, in a fash- 
ion, appropriating GPS technology for creative purposes. (...) Consequential 
financial, political and creative-capital investments are one of the drivers of 
interest in the digital territorialization of physical geography, thereby estab- 
lishing it as an interface for electronic media experiences. Through this ter- 
ritorialization, real-estate has become virtual-estate.”?9 

In 1996 Art+Com presented TerraVision®, a self-contained, Virtual Reality, 
1-to-1 representation of the Earth. A stylised globe was the interface for the 
audience to zoom in on any location in the world and obtain minutely detailed 
pictures. Ten years later this kind of system was online: In July 2005, the 
search engine Google, with “Earth”, created the possibility of a virtual world 
tour by satellite picture, with zoom function. Chip Online wrote: “The digital 
globe makes zooming from space to home town possible.”*! In this way, the 
filmic archetype “Powers of Ten” is now directly available for every computer 
user. The online globe offers a 3D map of many parts of the earth, in that it 
compiles satellite and aerial photography of towns and industrial areas. The 
victims of the hurricanes in New Orleans and elsewhere used “Google Earth” 
in order to form a picture of the destruction of their homes and neighbour- 
hoods. “Google Earth” picks up on the idea of scaling, in order - it appears — 
to get to grips with the world: regarding its weather systems, its economy and 
global totality. Satellite images send views from a great distance and create 
a distanced point of observation. These images, which seemingly extend our 
sight limitlessly, suggest definitions such as: total surveillance, total over- 
view. These are global positioning systems of complete visibility of time and 
space. 

29 Bleecker/Knowlton 2006 


30 The project was started by Uli Weinberg at Terratools. See <http:/ /www.artcom.de/ 
index.php?lang=en&option=com_acprojects&id=5&Itemid=144&page=6>. 


31 More about Google Earth in “Chip Online de” see <http:/ /www.chip.de/downloads/ 
cl_downloads_13015193.html>. 
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7. Presence: Models for Interactivity 


The idea behind the narrative form “film” is to represent events and tell 
stories arising from researching the realms of the psyche. Where film ends, 
the digital and interactive, process-related environment begins. Media artist 
Simon Penny (1995) differentiates the difference in perception of a painting, a 
film and of interactive media art as follows: “A painting is an instance of rep- 
resentation. A film is a sequence of representations. Interactive artworks are 
not instances of representation, they are virtual machines which themselves 
produce instances of representation based on real time inputs.” 

The theatre, with its abstracting stage and real-time input, appears a bet- 
ter role model for interactive concepts than film, with its ready-made images. 
“Home of the Brain” was conceived as a digital memory space and as a venue 
for battles of spoken words. The theatrical paradigm stands for the position- 
ing of information in space and for animating the audience. Nonetheless, 
visitors understand their virtual observer perspective in “Home of the Brain” 
as if they were walking through a film. This is because this virtual surround- 
ing could only be experienced as an individual with data glasses and data 
gloves. 

The return channel for the virtual meeting with others was technically 
only realisable a few years later with “Murmuring Fields”? (1998-2000), 
an audiovisual soundspace for several interactors on stage. We had built 
a shared environment for real-time interactive performance. So the virtual 
space became not only metaphorically virtual, but also physically real, an 
accessible and tangible sound. For “Murmuring Fields” we developed a mixed 
reality” method for the penetration and superimposition of physical and elec- 
tronic space. Data space and stage space are interconnected with one another 
and overlap by means of an optical tracking system. Two dancers utilise the 
interactive sound space of different sites. With their bodies in action, they 
employ the soundspace as an instrument. They play with words, images and 
sounds, and they oppose the system with their bodies in order to avoid being 
overwhelmed by technology. 

In “Murmuring Fields” digital information - sounds and figures - is located 
in the space as if the room were furnished with data.** Every movement of 


32 Murmuring Fields is documented on the online media art archive netzspannung. 
org <http://netzspannung.org/database/murmuring-fields> and on MedienKunstNetz 
<http: / /www.medienkunstnetz.de/works/murmuring-fields>. 


33 Mixed reality (MR) (encompassing both augmented reality and augmented virtual- 
ity) refers to the merging of real and virtual worlds to produce new environments and 
visualisations where physical and digital objects co-exist and interact in real time. See 
<http: / /en.wikipedia.org / wiki /Mixed_reality>. 


34 Strauss 1999 
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Fig. 2. Murmuring Fields - electronic multi user stage environment (1991) 
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the body is captured with an optical body-tracking technique.” Movement 
is transferred from real space into data space, and translated into a sound 
collage. Spoken texts are broken up into words and syllables. Movement in 
space creates movement in the text. Two interactors produce text samples 
by Joseph Weizenbaum, Marvin Minsky, Vilém Flusser and Paul Virilio — our 
keen thinkers from the earlier work “Home of the Brain”. “Poli-tic-tic-tic”, says 
Flusser’s voice as a performer bows backwards and forwards and thus inter- 
prets a part of Flusser’s words: “Youngsters at the terminals; they turn their 
backs to politics and turn to each other.” This passage, from an interview 


35  Body-Track is part of the eMuse-Systems, which were developed as a production 
system for “Murmuring Fields” (Strauss/Fleischmann 2007). 
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with Vilem Flusser in 1990 on Austrian television, was sampled and built into 
the sound database. The dancer triggers syllables with her body and forms 
speech. She plays with the meaning of the concepts. Text is translated into a 
texture of sound and movement. Theatre Scientist Martina Leeker sees that 
“in such information-technological modifications of theatre and stage (...) the 
internalisation of thought and imagination are overridden inasmuch as the 
actors become bodily involved. The externalisation of thought and imagina- 
tion through new media forms allows the mental ‘Mixed Reality Room’ to 
develop, and at the same to operate alongside our written cultural heritage.”°° 
In “Murmuring Fields” movement of the body moves sound and image. The 
art historian Oliver Grau judged that with “Murmuring Fields” a new type of 
space of mind had been created.’ 


8. Media Art as Public Thinking Space 


The question of how digital information can translate not only the meta- 
phoric virtual, but also the physically real, into accessible and understandable 
domains marked the passage to media architecture. By this we understand 
an architecture which connects people, space and data with one another. It 
creates an extended area of activity. Our first experiment interlinking data 
and virtual space resulted in “Home of the Brain”, an immersive environment 
for a single person. Connected to this work we developed “Murmuring Fields”, 
a sound environment for two performers on a public stage. With “Energie- 
Passagen”? (2004) we explored data connected to public space with an audi- 
ence passing by day and night. It was strolling through news: promenad- 
ing through information space. This model of public space was possible for 
us, following the inclusion of text and natural language, after our research 
and development of linguistic tools. The project “Energie-Passagen” is about 
reading and describing the city. It schematises public and private interest in 
information. This public installation offers the possibility of association, fil- 
tering and choice of a flow of words, which allow participant-oriented opinion 
formation and therefore a public interactive thinking space (Denkraum).*° 

The starting point of “Energie-Passagen” is texts from mass-media 
daily news. An automatic computer technique analyses the daily newspa- 
per and reduces it to the 500 most-used keywords. In this case it was the 
“Süddeutsche Zeitung”, which was analysed over four weeks by a semantic 
tool. The filtered terms appeared as a large screen-projected “flow of infor- 


A 


36 Leeker 2000 
37 Grau 2004 
38 <http://energie-passagen.de> 


39 The notion of the German Denkraum refers to Aby Warburg and Hanna A,rendt. 
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mation” in November 2004 in front 
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of the “Literaturhaus” in Munich. 
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Visitors could choose definitions on- 
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site and “interpose” them into the 
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flow. In this way text movements are 
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nections between the definitions to 
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emerge. The definition network cre- 
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as multi-voiced echo. In addition, 
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a world map visualises the journey, 
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which then takes the chosen defini- 
tion through the geographic land- 
scape of the news. The visible result 
of the visitors’ preferred words is the 
“Living Newspaper”. Its dynamically 
generated choice is projected within 
their original sentence onto the 
“information cube”. The deconstruc- 
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tion of the newspaper, which results 
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from the fragmentation of its origi- 
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Words in motions set the visitors’ 
own associations free. This unusual 
view of familiar daily news leads to 
surprising actions and reactions. Every day and evening of the four weeks, 
the on-site public discussed their own associations related to the daily news 
reports. An especially pleasing aspect was the discussions with older visi- 
tors, who felt themselves involved in the current developments and enjoyed 
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being able to experience something other than established forms of art. With 
“Energie-Passagen” we schematise speech in public space. The work shows 
how meaning can emerge through deviation, and offers the on-site public 
a sensual, tangible space for action, which sets free new potential for com- 
munication. Communicative performance and performative presence emerge 
through interactivity of audience and virtual space. The greater part of the 
approximately 4000 on-site visitors to the installation were between 50 and 
70 years old. During the one-month duration of the installation there were 
a further 3,000 internet visitors, some of whom also sent comments.*? More 
than four years later, in June 2008, there have been nearly 50,000 visitors 
who have seen the installation on the internet.*! The audience on site creates 
performative presence. The interactive spectator online creates a different 
form of performative presence with the web installation. 


9. Conclusion 


Film lives from its heroes, whilst the interactive players can themselves be 
actors and heroes - even in an art installation. Film needs an audience which 
gets inside a story and loses itself in it. This narrative art usually requires 
a plot, which includes a storyline and action. Normally, a plot is character- 
ised by a beginning, middle and an end. This rule does not apply to inter- 
active stories. Interactive media art assumes that observers are themselves 
active. Thus, the activity of participants can be interpreted as communicative 
performance. One’s own participation is a precondition for the experience of 
story and plot. 

The core of the interactive plot is not about the communication of one 
single reality, but lies in differing viewpoints and positions based on hyper- 
text structures and generative processes. The interactive performance is more 
about the volatile association of thoughts than the linear narrative of a film. 
As an analogy, one can describe the process of the exposure to interactive 
installations as thought in action and action in thought. This in turn indi- 
cates making visible the activity of thought processes. Communicative per- 
formance and performative presence evoke the atmosphere of the artwork as 
a thinking space (Denkraum). 


A shortened re-working from Medienkunst im Aufbruch - Geschichten erzählen 
von interaktiv bis reaktiv, ed. Klaus Rebensburg, Aachen: Shaker 2005, pp.65-84 


Deda 
40 Visitors’ comments: see <http://www.energie-passagen.de/presse2.html> (last 
access: June 2008). 


41 Energie-Passagen as a web installation: see <http://www.energie-passagen.de/ 


webinstallation> (last access: June 2008). 
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Prologue 


Recent diverse artistic projects using interactive technologies are indeed 
complex, but many of them together form a new direction, which relocates 
and dissolves traditional boundaries between different categories of art. This 
article is particularly concerned with a shift related to the concept of music. 
In the culture preserving the heritage of Western art music, music often refers 
to works of art consisting of well-formed sound structures which are free of 
any purpose, i.e. autonomous, and can be understood by distant attentional 
listening. This understanding of music as an autonomous work of art was 
supported by autonomous aesthetics grounded in romanticism on the one 
hand, and by the philosophy of history oriented towards the idea of progress 
of Western history towards the point of modern times on the other.! 

Even though metanarrations acting as a legitimation of Western art music 
began losing their validity at least due to the postmodern discourses on del- 
egitimation and plurality,? it is remarkable that discourses on legitimation of 
one category of music definitely decay in the digital era. It may be observed 
that there is an unexpected shift from a traditional concept of music, due to 
the essentially changed format of production and reception of “art” and of for- 
mation of artistic experience in New Media Art, even in the cases where sonic 
materials are primarily used and therefore auditory perception is strongly 
engaged. 


“Interactivity” as a key concept 
calling into question a traditional understanding 
of music in the Western music tradition 


“Interactivity” is a concept which came into focus in Western music tradi- 
tion through a dadaistic trend, in which John Cage is a key character.” This 
concept is closely related to questioning the traditional concept of music as 
an autonomous work of art which is considered as intentionally produced by 
a ‘genius’ on the one hand and as receptively perceived by the audience on 
the other. Cage aims at ‘indeterminacy’ of the relationship between composi- 
tion and realisation using a graphical, meaning-free notational system with 
which musical parameters are not determined as absolute variables, but in 
a relative relation to each other. He carried out the performances of happen- 
ings and fluxus arts with Merce Cunningham, Robert Rauschenberg, and 
1 Kim 2004, Chapter 3-4 
2 Lyotard 1979; Kim 2004, Chapter 6 


3 I refer to “Western music tradition”, since the concept of interactivity should be dis- 
cussed from a different point of view when taking into account other musical cultures. 
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David Tudor, among others, at the Black Mountain College in North Carolina 
(U.S.A.) at the beginning of the 1950s, reformulating the concept of art and 
the relationship between artist, work of art, audience/public and environ- 
ment: In these performances, creativity is not manifested in a durable work 
of art, but brings forth action. “Interactivity” becomes a core concept which 
makes a musical work emerge from the interplay between the composer, the 
performer, and the audience, from which an unpredictable effect arises dur- 
ing each processual performance. In this way, the artist becomes part of her 
or his work of art.* 

Cage opens a new conception of music. Music becomes an aesthetic 
experience of the unpredictable, the source of which can be found elsewhere 
than in the intention of a musician being considered as the subject of music. 
Musical performance is not conceived of as a stage on which an intentional 
subject expresses her- or himself, but as an occasional situation in which 
sounds come into the focus of attention. Such desubjectivisation gives rise 
to the demise of the progress idea of musical structure underlying Western 
art music, which is oriented towards the modern philosophy of history, and 
furthermore to the abolition of the category of the ‘closed’ work of art into an 
open aesthetic process. 

Cage’s musical compositions include diverse experiments with musi- 
cal materials, instruments, and performance constellations. However, it is 
remarkable that he is one of the pioneers in new directions of music com- 
posed electronically.” In the 1930s and 1940s he used a film phonograph and 
electronic musical instruments (e.g. “Novachord”, a polyphonic synthesiser 
manufactured by Hammond, and “Theremin”, an antennae-based musical 
instrument played with free hand gestures) for The Future of Music: Credo 
(1937) and oscillators, turntables, and generators for his composition series 
Imaginary Landscape No. 1-4 (1939-1951). The use of sound materials cre- 
ated by electronic means and the exploration of different electrotechnical 
methods of sound collection, generation, and control are closely related to his 
search for a new concept of music. 

In the 1950s and 1960s, after he introduced the principle of “indetermi- 
nacy”, a live character was assigned to his compositions for sounds created 
electronically, taking the process of electronic sound generation out of the 
studio. A musical event of electronic sounds which is formed as composi- 
tion results from a series of actions which can vary in each performance. A 
predetermined compositional idea is only concerned with the whole struc- 
ture including duration, possible actions, sound materials prepared, and if 
4 Haris1987 O T 


5 Since the term “Electronic Music” is generally assigned to the musical genre in 
which a sine wave generator has been used for sound generation in Cologne since the 
1950s, I avoid applying this term to early compositions of Cage. 
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necessary a kind of dramaturgy. The completion of this composition relies 
rather on (partially unpredictable) performative actions. Variations V (1965), 
which Cage realised with Merce Cunningham as an interactive composition 
for dancers, can be taken as an example. The performance stage for dancers — 
equipped with antennae measuring the electromagnetic capacities produced 
by the various distances from the dancers to each antenna, and with photo- 
electrical cells measuring light conditions on the stage changed by the danc- 
ing movements — serves as a stage for actions which lead to musical inter- 
action between pre-recorded sound materials on tape recorders and short 
wave radios resulting in a sound mix. With the vertical movements of their 
bodily parts, the dancers were capable of influencing the sound intensity; 
the horizontal direction of dancing movements led to varying proportions of 
sound distribution on the different amplifiers. In his Remarks 37 (1965), Cage 
makes it clear that the composition of Variations V consists in the medial con- 
figuration, i.e. elements of sound system and control units.® 


The early stage of interactive live electronic music 


The experiments with live electronic music which Cage’s new concept 
of music and musical composition underlies, however, seem to derive from 
the main trend of live electronic music, taking into account, in particular, 
the European scene of live electronic music. Even though an integration of 
live music performance into concerts of Electronic Music is also an essential 
aspect of the latter, the idea of interactivity which Cage explicitly deals with, 
questioning the concept of Western art music as autonomous work of art, is 
hardly found in the early stage of live electronic music rooted in European 
Musique Concrete and Electronic Music.’ This might consist in the fact that a 
number of the composers of Musique Concrete and Electronic Music tend to 
extend new musical materials and electrotechnical procedures to create an 
autonomous musical piece, following the tradition of Western art music. 

In most performances of live electronic music, composers tried to experi- 
ment with the possibilities of integrating live performing musicians (sing- 
ers and instrumentalists) into the performances of electronic music, which 
otherwise are purely based on the reproduction of a pre-composed piece via 


zz 


6 Cage 1965 


7  TheFrench sound engineer and composer Pierre Schaeffer founded Musique Concrète, 
which is created with “concrete” sound objects by an electronic means. Everyday noises, 
speech sound, or acoustic sounds are recorded using a microphone on magnet tape and 
processed using montage, filtering, mix, and transposition etc. (Schaeffer 1966). The 
centre for Electronic Music in Cologne was engaged in techniques of sound generation 
in a purely electronical way. Sinus tones were generated by an generator, and then either 
overlapped or mixed with filtered noises and impulses and recorded on a tape (Manning 


285 


loudspeakers. In this way, they aimed at the interplay between traditional 
(live) music performance and reproduction of electronic musical composi- 
tions. Live electronic music can in this context be seen rather as a comple- 
mentary approach to traditional electronic music performances, giving them 
alive stage character and enabling the use of an ensemble of instrumental (or 
vocal) and electronic music. A relocation of the roles of composer, performer, 
and the audience or the concept of music as a work of art, however, does not 
seem to be a topic in the European trends of live electronic music. 

Interactive live electronic music has been developed in the context of 
live electronic music as an approach to solving the lack of context-sensi- 
tive variations of usual live electronic music consisting of a live perform- 
ance of instrumental or vocal music accompanied by pre-composed, fixed 
tape music. Interactive live electronic music tries to render electronic music 
parts capable of reacting to live performance situations and varying context- 
sensitively according to each performance situation. For this purpose, a com- 
puter-aided analysis of information coming from live performance - utilising 
interactive software - serves as a basis for an ‘interactive’ output of electronic 
sounds. “Interactivity” is here related to the capacity of the computer system 
to “change [its behaviour] in response to musical input”, as the composer 
and researcher of computer music Robert Rowe defines “interactive music 
systems”.® This is similar to a technical concept of interactivity used in early 
information technological research on human-computer interaction (HCI). In 
those so-called interactive music systems, the tasks of the computer consist 
in an interpretation of raw data captured during a live performance and in 
their use for musical composition and for sound generation. In the early stage 
of live electronic music, the role of interactive music systems was computer- 
aided automatic accompaniment of a live performance. The composer Joel 
Chadabe who introduced the term “interactive composing” in 1983 gives an 
overview of an (early) interactive music system (Fig. 1). 

In most performances of early interactive live electronic music, musical 
information — e.g. pitch, loudness, dynamics etc. — served as the input data 
of interactive music systems. The so-called score following techniques were 
developed for this purpose:? A musical score for the live performance is put 
into a computer system in a certain form. The live performance is captured 
via a microphone or a MIDI interface and analysed by the computer system in 
real-time. The analysed sound events are compared to the score stored in the 
computer. If there is a match, the computer accompanies the live perform- 
ance, generating sound events algorithmically — based on the score storage. 


sg zZ 


8 Rowe 1993, p. 1; This book entitled “Interactive Music Systems” is the first book 
dealing with those systems and Interactive Music systematically. 


9 Vercoe 1984; Dannenberg 1984 
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Fig. 1. The general principle of procedure of an interactive 
music system (Chadabe 1983) 


Most score following techniques, however, are based on the principle of 
a knowledge-based system developed by a traditional approach of artificial 
intelligence. A musical score, which is put into a computer system, acts as a 
kind of represented knowledge. A score-following technique, which allows the 
computer system to monitor input events coming from live performances of 
an instrumentalist and to compare these with the knowledge - the score — of 
the computer system so as to process computer-generated sound parts, has 
a hierarchical structure of interaction processes - from the sensing up to the 
processing and down to the response stage.'!° A knowledge-based process of 
interpretation of information coming from the sensing stage takes place in the 
processing stage, which is separated from the sensing and response stage. 
In other words, an exchange between internal and external processes does 
not take place during the processing stage. Output events of machines as a 
response to input events are determined in this isolated stage and realised by 
top-down organisation. Hence, knowledge-based interactive music systems 
are conceived of as decoupled from the environment and therefore as not truly 
interactive. What is realised in the early form of interactive live electronic 
music is a more flexible accompaniment of electronic music generated by 
algorithms to a live instrumental or vocal performance. In most cases, the 
traditional concept of music as “work of art” still remains. 


Bodily-based interaction with sound events 


Contrary to score following techniques, which are almost exclusively used 
for interactive live electronic music originating from the tradition of Western 
art music, further techniques of motion tracking are applied in broad artis- 
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10 Rowe 1993 
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tic contexts. Since the 1980s, in which technological possibilities to capture 
bodily actions and to use them as input data for the computer system were 
standardised, many artists have increasingly experimented with a variety of 
interactive art based on bodily actions of an observer/performer, whether 
interactive sculpture, installation, or performance. 

In interactive live electronic music, motion tracking techniques, which 
became the focus of attention in the 1980s, have met with large interest since 
the 1990s, so as to lead to diverse experiments from both artistic and infor- 
mation technological perspectives. Motion tracking techniques are combined 
with so-called gesture mapping, which means an effective computable map- 
ping from bodily gestural parameters (e.g. position of a body part or intensity 
of finger pressure) into parameters for sound synthesis (e.g. frequency or 
amplitude of acoustic wave form). Hence the development of different strate- 
gies of gesture mapping has become a hot issue of information technological 
research on interactive live electronic music. Most of gesture mapping, how- 
ever, consists of ad-hoc solutions, not based on a general rule. Gesture map- 
ping, however, seems to offer diverse possibilities of designing the interactive 
relationship between bodily actions and sound events. 

Musical interaction which is not based on score-following techniques, but 
on strategies of gesture mapping, does not only behave as an “ensemble” (live 
performer and computer accompaniment). Some modes of musical interac- 
tion can also be described with further metaphors 
inspired by traditional music practices such as 
playing an instrument or conducting. A multiplic- 
ity of musical interfaces simulating, extending or 
re-configuring traditional musical instruments 
have been used in interactive live performances 
in which a performer has the role of a player of 
this new “instrument” and the computer acts as 
a musical instrument.!! The definition of a rule 
of gesture mapping is a main task of composi- 
tion, which however is often not decoupled from 


performance. Therefore most composers act at 


Fig. 2. Max Mathews 
with the Radio Baton/ 


the same time as a performer “playing” her or his 


A musical interface designed especially for her or 
Drum in 1992. Photo by 


Patte Wood (Chadabe his compositions, which can be realised and com- 
1997, p. 231) pleted during the process of performance. 
E Some interfaces allow the performer to act as 
a conductor shaping a musical composition expressively. For example, the 


Radio Baton/Drum developed by Max Mathews (Fig. 2) renders the performer 


LEE 
11 Editorial footnote: see for a few examples Jäger/Kim and Goto for an example of his 
violin interface SuperPolm in this book. 
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capable of controlling musical expressiveness during the performance, mov- 
ing two batons equipped with different radio emitters over a square surface 
equipped with receivers. Each baton provides information about its horizontal 
(x) and vertical (y) position and its height (z). In this way, a three-dimensional 
movement of each baton can be followed and mapped into parameters for 
digital sound manipulation.'? 

Such modes of musical interaction inspired by traditional music practices 
give rise to a rethinking of principles of bodily-based musical interaction, 
which is basic for each music performance. Especially the coordination of 
auditory and tactile sensations has become the focus of newer research on 
the design of musical interfaces based on approaches of physical, haptic, 
and tangible computing.’ A number of so-called haptic musical interfaces 
are capable of offering haptic feedback so that a user/performer can touch, 
press, or pull a physical material to enter into musical interaction mediated 
by algorithmic computation. A group of tangible interfaces can be grasped, 
squeezed, or moved from one place to the other so that a user/performer can 
use physical actions with the help of physical objects directly situated in a 
real environment to control and represent digital information.'* Furthermore, 
force feedback or vibrotactile feedback have been additionally simulated in 
some musical interfaces in order to improve musical interaction with new 
interfaces. Usually a user/performer dealing with haptic or tangible musical 
interfaces is requested to concentrate on physical actions which are rarely 
guided by visual representation. Hence she or he can develop the skill of coor- 
dination between tactile and auditory feedback the computer system offers, 
which is similar to the experience underlying the playing of traditional musi- 
cal instruments. In this way, music can be dynamically composed, “feeling” it 
at the same time during the bodily-based interactive performance guided by 
a loop of double feedbacks. 

Some strategies of gesture mapping enable a kind of dance-music inter- 
action. The whole bodily movement of a performer can be tracked to trigger 
and control sound events. Dancing here no longer means an adjustment to 
pre-composed music, but a process of composing and modifying a musical 
structure which adjusts to dancing movements. A choreography of dance 


020 za 


12 Boulanger/Mathews 1997 


13 Igoe/O’Sullivan 2004, Brewster/Murray-Smith 2001; McGookin/Brewster 2008; 
for information about current research on tangible computing see Proceedings of the 1st 
International Conference on Tangible and Embedded Interaction 2007, online available: 
<http: / /portal.acm.org/toc.cfm?id=1226969&type=proceeding>; Proceedings of the 2nd 
International Conference on Tangible and Embedded Interaction 2008, online available: 
<http:/ /portal.acm.org/toc.cfm?id=1347390&idx=SERIES11433&type=proceeding&col 
1=ACM&dl=ACM&part=series&WantType=Proceedings&title-TEl&CFID=17833818&CFT 
OKEN=87925403> (last access: March 2008). 


14 Editorial footnote: see for some examples of tangible musical interfaces Weinberg in 
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serves here not as an interpretation of musical composition, but rather as a 
main part of musical composition which can only be realised by dancing per- 
formance. A complete composition which can be reproduced independent of 
each performance does not exist. Experimental computer-aided dance-music 
interaction is a new field which dissolves traditional categories of dance and 
music. Besides camera tracking techniques, some wearable interfaces such 
as the DIEM digital dance system by the researchers at the Danish Institute 
of Electroacoustic Music, the MIDI dancer by the artists group Troika Ranch 
and the SSPeaPer (Sensor/Speaker Performance Interface) by the composer 
and researcher Curtis Bahn have been developed especially for the purpose 
of dance-music interaction. 

The principle of dance-music interaction also underlies many interactive 
sound installations which are not from a stage-oriented performance genre, 
but originate from the category of fine arts. Interaction often takes place in 
this context involuntarily, such as by entering into an installation room and 
triggering a sound generator. Further modes of interaction with sound events, 
however, can be actively explored. This act of interactive exploration may be 
compared to dancing. For instance, the Very Nervous System developed by the 
Canadian media artist David Rokeby is used both for dance-music interac- 
tions and for interactive sound installations. The Very Nervous System, which 
is based on a camera-tracking technique, is a sonically oriented system, since 
a virtual environment designed by Rokeby does not provide a visual repre- 
sentation, but consists solely of sound events. Therefore an observer cannot 
remain passive in order to enter into a computer-generated world, but attains 
an artistic experience only through an active improvisation. Rokeby intended 
to develop an improvisation system with which an observer/performer can 
explore an interactive relationship between her or his dancing actions and 
the sonifying installation environment.!? Rokeby’s Very Nervous System views 
each movement not as an individual static image, but as a movement flow 
in the context of linear movement sequence. The temporal aspect comes in 
via the computation of movement analysis. The movement of the observer/ 
performer is interpreted in a horizontal linear flow, so that a certain move- 
ment can be transformed into different sound events according to the whole 
movement context. The interaction that the Very Nervous System provides is 
not based on the metaphor of dialogue which underlies the most traditional 
approaches of HCI. According to Rokeby, a dialogue implies a separation of 
functions of perception and reaction.'® His system, however, organises per- 
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Fig. 3. Very Nervous System. Courtesy of David Rokeby 
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ception and expression at the same time, so that the observer and the com- 
puter system form a loop of feedback which is very close and complex.!7 


Interactive emergence 


The concept of interactivity has been a topic which in turn needs rethink- 
ing due to the recent approaches of information technological research, arti- 
ficial intelligence, and cognitive science, which place emphasis on new para- 
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digms questioning traditional concepts and their relations (e.g. perception, 
cognition, action). New technological methods which have been developed in 
the course of paradigm shift flow into New Media Art, which acts partially as 
an experimental environment for theoretical questions. 

Aritificial Life (A-Life) Art, for instance, is a typical field which was insti- 
gated utilising a newer approach of artificial intelligence, A-Life procedures.'® 
A-Life uses the concepts of information processing and computational mod- 
eling to understand life in general.'® A-Life research aims at the definition of 
simple rules from which a complex behaviour emerges” — contrary to tradi- 
tional approaches of artificial intelligence, which give a machine a task to be 
solved, writing a program accordingly so that the machine can execute this 
task. Hence A-Life procedures focus on simple processes interacting among 
each other and in this way generating a high-order system behaviour.?! A-Life 
Art is based on a procedure to generate living behaviour of artificial agents, 
which is characterised as a bottom-up approach. An A-Life approach of com- 
putational modeling has recourse to the biological nature of creatures, so that 
properties such as self-organisation, emergence, reproduction, and adaption 
are assigned to a machine. Artificial Life (A-Life) Art accordingly experiments 
with different visual, sonic, or physical agents (e.g. robots) which show an 
emergent behaviour. 

In communities of computer music research, a discourse on live algo- 
rithms for music (= LAM) has very recently been instigated, which is also 
the title of a series of conferences taking place since 2004. The main interest 
is directed towards autonomous interactive algorithms that are character- 
ised by “adaptation and creative contributions of algorithms to the musical 
dimensions of sound, time and structure.””? It is concerned with interac- 
tive aspects of algorithms inspired by swarm intelligence, evolutionary com- 
putation, artificial life and complex dynamics. Live algorithms are intended 
to avoid “systems pre-loaded with syntax derived from music theory” and 
“rule-based approaches that relate input to output in a simple way.”?? Some 
composers and media artists who are engaged in interactive composition and 
improvisation up to now have experimented with live algorithms approaches 
in which interactivity is characterised by emergence. 

The interactive sound installation Natural Selection (2005) by Tom Davis 
and Pedro Rebelo can be taken as an example of LAM projects. Davis and 
Rebelo use ten mechanical “sound objects”, each of which consists of a reso- 


BZ 20zz22 za 


18 Bird/Webster 2001; Sommerer 2001; Wilson 2003; Whitelaw 2004 
19 Boden 1996, p. 1 

20 Boden 1996, pp. 3-4; Braitenberg 1984 

21 Boden 1996, p. 4 

22 See <www. livealgorithms.org> (last access: March 2008). 


23  <www. livealgorithms.org> 


292 


nator driven by a motor. Interaction takes place both among these sound 
objects and between them and the observer. Natural Selection is based on an 
algorithm which is inspired 
by frogs’ behaviour, espe- 
cially a female frog’s choice 
of her mating partner out 
of the calling chorus of 
male frogs.” Davis and 
Rebelo summarise rel- 
evant properties of frogs’ 
mating calls from current 
research results and model 


them in sonic behaviour 
of the installation Natural 


Fig. 4. The installation Natural Selection 
Selection: The dominant call 


(2005) at the Música Viva-Festival in Portugal. 
frequency is related to the Courtesy of Tom Davis 


: PA 
size of the frog, the pulse 


rate to the temperature of the environment, and call rate and duration to 
the preference of each individual creature.” Natural Selection uses a simple 
model of interactions between male frogs in a chorus, which are symbolised 
as sound objects, while the observer has a role of the female frog (Fig. 4). 
The researchers from the University of Tokyo Jean-Julien Aucouturier, 
Yuta Ogai, and Takashi Ikegami have very recently experimented with a tech- 
nique to make a robot dance to music autonomously and synchronously. 
They avoided a pre-programming of dance patterns. Instead, they built basic 
dynamics into the robot which render it capable of developing emergent behav- 
iour. The dance movements of the robot were controlled by motor commands 
generated by using an artificial neural network (ANN), a network of artificial 
spiking neurons, each controlled by a biologically-inspired model (FitzHugh- 
Nagumo (FHN)).76 A sequence of pulses detected from the beats of the music 
was processed by this ANN, and the output of the FHN network was mapped 
into the sequence of pulses being used for the robot dance corresponding 
to the beats of the music. Although this project has not been applied in an 
artistic context yet, the increasing number of robotic art and musical robotics 
projects indicates the current directions of using robots as agents of A-Life, 
contrary to traditional approaches of robotics based on top-down rules.” 


A 


24 Davis/Rebelo 2005 
25 Davis/Rebelo 2005, section 2 


26 Aucouturier/Ogai/Ikegami 2007, available online: <http:/ /www.jj-aucouturier. 
info / papers /ICONIP-2007.pdf> 


27 For detailed discussion on musical robotics projects see Seifert /Kim 2007, 2008. 


293 


Christoph Lischka’s project par_cho |r (2001-2004), which exists in differ- 
ent implementations such as par_cho |r : mono, par_cho |r : fugue (as perform- 
ances) and par_cho|r : trans (as an installation), deals with sound-generat- 
ing algorithms embodied in the form of a ball 
robot. In performance projects, ahuman bass 
clarinettist interacts live via a ‘sound lan- 
guage’ with a ball robot which ‘listens to’ and 
‘analyses’ music played on a contrabassoon 
and acquires in this way some kind of ‘hearing’ 
knowledge.”® Accordingly this ball robot moves 
within a certain defined space and plays a con- 
tra part. What is observed is an improvisation 
of two “musicians”, which becomes evident 
in the emergent musical structure.” Lischka 
describes the project as following: The project 


deals with “artificial ecologies, i.e. artificially 
Fig. 5. par_cho|r : fugue. generated ‘creatures’ with their ever particular 
Performance at the Inter- environments.” “An artificial lived-in world of 
national Symposium on 
Music, Art, and Robotics 


(SMARt) in Bremen (14 June 
2006). Courtesy of and ultrasonic sensor and actuator technolo- 


‘ball creature’ is created through simulation 
and reconstruction of this organism by sonic 


Christoph Lischka gies. This ‘ball creature’ develops in each indi- 
Sea aa vidual and collective characteristic »swarm«) 
sonic, visual and choreographic interaction patterns which become in turn 
an object of artistic experience and performance.”*° 

Such algorithms which allow a machine or/and a unity of human and 
machine to display emergent behaviour become an essential means of creat- 
ing works of New Media Art. An experience with evolving sonic behaviour 
related to her or his behaviour makes an observer act as an actant?' who, 
however, does not always possess agency, but is affected at the same time 
as a patient. A true interaction is based on oscillation between agency and 
patienthood among the actants participating in this interaction, in a continu- 
ous circle of affecting and being affected, in short: reciprocal turn-taking. 
This concept of interactivity which allows an observer to shape a dynamic 
process of artistic creation leads to a calling into question of the connota- 
tion of “music”, which is not limited to a concert hall for Western art music 
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or opera house culture demanding only one musical behaviour: a passive 
distant behaviour of music listening. “Interactivity” as a main subject in New 
Media Art which is of great interest in our digital era makes clear the neces- 
sity to delegitimate the hegemony of Western art music and at the same time 
to rethink the nature of music, which may be conceived of as interactive from 
the outset. 
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1. Introduction 


It is widely perceived that the computer has enriched and advanced the 
art form of music. Digital technology brought new palettes of sounds, compo- 
sition techniques, and production methods; innovations in digital compres- 
sion and distribution changed music consumption and listening practices; for 
performers, novel musical instruments and controllers have been developed 
based on a variety of sensing, interaction, and mapping approaches. But after 
more than two decades of research in computer music, a fundamental ques- 
tion must be asked - has digital technology truly innovated and enriched the 
expressive, emotional, and creative core of the musical experience? It is not 
clear that the answer to this question is as positive as we music technologists 
would like to think. 

During the last ten years, inspired and motivated by the prospect of inno- 
vating the core of the musical experience, I have explored a number of research 
directions in which meaningful use of digital technology bears the promise 
of revolutionising the medium. The research directions identified — gestural 
expression, collaborative networks, and constructionist learning — can lead 
to musical experiences that cannot be facilitated by traditional means. The 
first direction builds on the notion that through novel sensing and mapping 
techniques, new expressive musical gestures can be discovered that are not 
supported by current acoustic instruments. Such gestures, unconstrained by 
the physical limitation of acoustic sound production, can provide infinite pos- 
sibilities for expressive and creative musical experiences for novice as well as 
trained musicians. The second research direction utilises the digital network 
in an effort to create new collaborative experiences, allowing players to take 
an active role in determining and influencing not only their own musical out- 
put but also that of their co-performers. By using the network to interdepend- 
ently share and control musical materials in a group, musicians can combine 
their musical ideas into a constantly evolving collaborative musical activity 
that is novel and inspiring. The third research direction utilises construction- 
ist learning, which bears the promise of revolutionising music education by 
providing hands-on access to programmable music making. Through interac- 
tion with physical computational objects, learners can construct personally 
meaningful musical artifacts that enhance and deepen their learning. 

While facilitating novel musical experiences that cannot be achieved by 
traditional means, the digital nature of these research directions often leads 
to flat and inanimate speaker-generated sound, hampering the physical rich- 
ness and visual expression of acoustic music. In my most recent work, there- 
fore, I attempt to combine the benefits of digital computation and acoustic 
richness, by exploring the concept of “robotic musicianship”. I define this 
concept as a combination of musical, perceptual, and social skills with the 
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capacity to produce rich acoustic responses in a physical and visual manner. 
The robotic musicianship project aims to combine human creativity, emotion, 
and aesthetic judgment with computational capabilities, allowing human and 
robotic players to cooperate and build off one another’s ideas. A perceptual 
and improvisatory robot can best facilitate such interactions by bringing the 
computer into the physical world both acoustically and visually. 

In this paper I will describe my projects portraying a musical journey that 
was initiated by my interest in extending acoustic music with digital technol- 
ogy and reached its most recent period by investigating the enhancement of 
digital music through physical-acoustical means. Each station in this jour- 
ney presents a different set of novel expressive and creative possibilities along 
with a set of limitations and constraints imposed by technology. 


2. Related Work, Goals, and Challenges 


The field of New Interfaces for Musical Expression! has received signifi- 
cant interest in recent years as researchers and musicians explore new sens- 
ing techniques, design approaches, mapping schemes, and sound generation 
methods to enhance and enrich musical expression. Research in this area can 
be categorised into two main areas — Imitated and Augmented Instruments, 
and Alternate Controllers. Building on the vast repertoire of familiar musi- 
cal gestures, researchers have created imitated and augmented versions of 
traditional instruments such as percussions, strings and woodwinds, among 
others. Alternative ways to play music have also been explored by using vari- 
ous sensing and mapping techniques such as in non-contact instruments 
wearable music and alternate tangible controller. Most of these instruments, 
however, have been created for particular compositions (usually by the inven- 
tor) and have been effective only within specific aesthetics boundaries. Only 
few controllers have shown durability and adaptability to multiple composi- 
tions in a variety of musical styles. Inspired by the tradition of great versatile 
acoustic instrument such as the piano, one of the main goals of my work was 
to develop controllers that are durable, versatile, and adaptable to multiple 
compositions, styles, and playing techniques. 

The second area of related work is in the field of Interconnected Musical 
Networks (IMNs) - live performance systems that allow players to influence, 
share, and shape each others’ music in real-time. Such systems, whether 
they operate in one physical space or over a wide-area network, provide an 
interdependent framework that can lead to rich social and musical experi- 
ences that are not supported by traditional group play. The development of 
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IMNs since the 1950s has been connected to the development of technological 
innovations — from John Cage’s early experimentations with interconnected 
transistor radios through the use of networked PCs by groups like the League 
of Automatic Music Composers and the Hub, to the current proliferation in 
collaborative Internet music. These experiments, however, usually require 
advanced musical skills and understanding by players and audiences, and 
often lead to inaccessible “high art” musical outcome. More recent collabo- 
rative musical installations for novices on the other hand, tend to simplify 
the musical experience for novices and are not geared to interdependently 
connect between novices and professionals. To address this gap, my work 
attempts to explore novel interdependent musical interactions that would 
provide both novices and experts with rich and inspiring, yet intuitive and 
easy to follow, collaborative musical experiences. 

The educational goal of my research is informed by related work in the 
field of constructionist learning. The constructionist approach emphasises 
the unique ability of digital technology to provide personal and configurable 
learning experiences to a wide variety of learners. The approach was con- 
ceived by Seymour Papert, who demonstrated how learning is most effective 
when students construct personally meaningful technological artifacts. Other 
researchers have elaborated on Papert’s ideas, showing how interaction with 
digital physical objects enhances children’s and adults’ learning. In music, 
however, little has been done to develop constructionist systems that attempt 
to connect between figural expressive musical experiences and formal aspects 
of theory and technique. In conventional music education systems, when 
music students are introduced to formal theory, certain important expressive 
aspects that came naturally in the early figural mode are temporarily hidden 
when learners try to superimpose analytical knowledge upon felt intuitions. 
My work attempts to utilise constructionist-learning methods to bridge the 
gap between the figural and formal learning modes through hands-on inter- 
action with programmable musical controllers. 

And lastly, I introduce the concept of robotic musicianship, taking up 
Rowe’s concept of machine musicianship. In this research area, scholars 
develop interactive systems that analyse, perform, and compose music with 
computers based on theoretical foundations in fields such as music theory, 
computer music, music cognition, and artificial intelligence. Several effective 
approaches for the design of such interactive musical systems have been 
explored over the years by researchers and musicians such as Dannenberg’, 
Cope’, Lewis*, Pachet®, and others. Such digital interactive systems, however, 
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are limited by the inanimate and flat nature of their digital musical repro- 
duction. Current research directions in musical robotics, on the other hand, 
focus mostly on sound production and rarely address social aspects such 
as listening, analysis, group improvisation, or collaboration. Both “robotic 
instruments” — mechanically automated devices that can be played by live 
musicians or triggered by pre-recorded sequences — and “anthropomorphic 
robots”’ - hominoid robots that attempt to imitate the action of human musi- 
cians — function mostly as mechanical apparatuses that follow deterministic 
rules. The motivation for establishing the field of robotic musicianship is to 
develop robots that can produce rich acoustic sound and visual cues, while 
utilising computational power and techniques of machine musicianship that 
are not possible with traditional acoustic instruments. 


3. The Projects 


3.1 The Musical Playpen (1997-1998) 


The Musical Playpen was the framework for my preliminary experimenta- 
tion with gestural musical interaction in a constructionist-learning environ- 
ment. The instrument was designed for toddlers and infants in an effort to 
explore whether very young children can participate in a meaningful, active 
musical experience. The environment 
allows young children to control two high- 
level musical aspects — contour and rhyth- 
mic stability - in an environment which 
is both familiar and fun: a 1.5-x-1.5-m 
playpen filled with 400 colourful plastic 
balls (Fig. 1). The playpen was designed to 


generate musical responses in correlation 
to children’s activity. Players’ movements 


Fig. 1. A child playing in the 
Musical Playpen around the playpen propagated from ball 
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to ball and triggered four piezo-electric 
sensors that were hidden inside four balls, one in each corner of the playpen. 
The balls’ ability to transmit hits to neighboring balls, combined with the 
sensors’ high sensitivity allowed for almost any delicate movement around 
the playpen to be captured by at least one sensor. The analog signal was then 
digitised and sent to a Macintosh computer running Max/MSP where it was 
mapped to musical output played from speakers below the playpen. Two oppo- 
site corners were mapped to control the melodic contour of an Indian raga, 
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so that the more energetic the players’ movements in these corners were, the 
higher the played Indian raga pitches became. Children could therefore cre- 
ate melodic phrases and manipulate their curves by changing the intensity of 
their body movements in these corners. Player’s physical activity in the other 
two corners were mapped to an algorithm that controlled the tempo, rhythmic 
variation, and timbre of percussive sequences in an effort to provide access to 
controlling rhythmic stability. The more energetic the players were near these 
corners, the more versatile and uneven the rhythmic values became. The 
tempo curve also fluctuated more sharply, as did the rate of timbral change. 

A number of observation sessions were conducted with the playpen at 
MIT and at the Boston Children’s Museum from 1998 to 1999. These sessions 
have shown a wide range of responses to the environment and the high-level 
musical control that it offered. For example, a 1-year-old infant started her 
session by triggering a sequence of notes as she was placed near one of the 
melodic curve corners. The infant looked in the direction of the sound source 
and tried to move her hand towards that corner, seemingly trying to repeat 
the music she heard. When she succeeded and another melodic phrase was 
played, she smiled, took one ball and tried to shake it, obviously without 
audible results. Frustrated, she then threw the ball towards a rhythmic cor- 
ner, generating a short percussive sequence. She approached this corner 
while moving her torso back and forth, laughing when discovering that her 
movements controlled the music. After a short break the infant started to 
move her body again back and forth, gradually accelerating her movements, 
generating less and less stable percussive sequences. Only after repeating 
this behaviour in another corner did the infant seem to be ready to use more 
expressive, less restricted gestures all over the playpen. 

These responses can indicate that with the right instruments and con- 
trols, young children can have access to spontaneous, expressive music- 
making as well as to more serious and thoughtful musical explorations. 
These findings encouraged me to develop a new set of instruments, which I 
entitled “The Squeezables”, in an effort to continue and develop models for 
high-level musical control, and to explore novel methods for networked group 
collaboration with older players, who can express and discuss their impres- 
sion of the experience. 


3.2 The Squeezables (1998-1999) 


In the Squeezables project, I attempted to add the concept of musical 
networks to my initial interest in gestural controllers and constructionist 
education. The goal of the project was to allow a group of players, novices 
and proficient musicians, to interdependently collaborate in constructing a 
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meaningful musical composition using unconventional expressive gestures. 
The instrument consisted of six squeezable and retractable gel balls mounted 
on a small podium, which players could simultaneously squeeze and pull to 
manipulate a set of low- and high-level musical percepts. The combination of 
pulling and squeezing allowed players to utilise familiar and expressive ges- 
tures and to control multiple synchronous and continuous musical param- 
eters. Several materials were tested and for the final prototype, soft gel balls 
were chosen, which proved to be robust and responsive, providing a sense of 
force feedback control that derived from the elastic qualities of the gel. Buried 
inside each ball was a 0.5-x-2.0-cm plastic block covered with five pressure 
sensors, protected from the gel by an elastic membrane. The analog pressure 
values from these sensors were transmitted to a digitiser and converted to 
MIDI. Pulling gestures were sensed by six 
variable resistors installed under the table. 
An elastic band connected to each ball 
added opposing force to the pulling gesture, 
helping to retract the balls back onto the 
tabletop (Fig. 2). 

In an effort to evaluate the high-level 


algorithms in the instrument, a number of 


Fig. 2. Three networked play- 


straightforward mappings were designed to 
ers play The Squeezables 7 y 
uuumuunuuunnunuuuemenuue COMNtrol relatively low-level musical param- 

eters. For example, one of the balls formed 
a one-to-one connection between squeezing and pulling gestures to the 
modulation rate and range of two low-frequency oscillators, respectively. For 
other balls higher-level algorithms were developed to control percepts such 
as contour and stability. For example, pulling and squeezing gestures of the 
“Arpeggiator” ball controlled a combination of musical parameters including 
tempo, pitch commonality, dissonance and rhythmic variation, so that the 
more the ball was squeezed and pulled, the more unstable an arpeggiated 
sequence became. To facilitate a coherent hierarchical interconnected inter- 
action, the balls were divided into five accompaniment balls and one melody 
soloist. The five accompaniment balls provided players with autonomous 
control - no input from the other balls influenced their output. However, 
these balls’ output was mapped not only to the accompaniment parameters 
but also to transform the sound of the “melody” ball. While pulling the “mel- 
ody” ball manipulated its own contour so that the higher it was pulled, the 
higher the melodic curve became. The actual pitches, as well as the MIDI 
velocity, duration and pan values, were determined by the level of pulling 
and squeezing of the accompaniment balls. This allowed the accompaniment 
balls to “shape” the character of the melody while maintaining a comprehen- 
sive scheme of interaction among themselves. 
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To experiment with these mappings I composed a short piece for three 
players. The piece, which was featured in Ars Electronica 20008, starts with 
a high-level of instability and builds gradually towards a repetitive rhythmic 
peak. Special notation was created for the piece — two continuous graphs 
were assigned to each one of the six balls. One graph indicated the level of 
squeezing over time and the other indicated the level of pulling. The proc- 
ess of writing and performing the piece served as a useful tool for evaluat- 
ing the mapping and sensing techniques used. In addition, discussions were 
held with novices and professionals who played the instrument. In general, 
children and novices were more inclined to prefer playing the balls that pro- 
vided high-level control such as contour and stability. They often stated that 
these balls allowed them to be more expressive and less analytical. Proficient 
musicians, on the other hand, often found the high-level control somewhat 
frustrating, because it did not provide them with direct and precise access 
to specific desired parameters. Some experts complained that their personal 
interpretation of the high-level controllers for stability differed from the one 
implemented in designing the instrument. Both novices and professional 
players found the multiple-channel synchronous control expressive and chal- 
lenging and the pulling and squeezing gestures comfortable and intuitive. 

These gestures allowed delicate and easily learned control of many simul- 
taneous parameters, which was especially compelling for children and nov- 
ices. The organic and responsive nature of the balls was one of the features 
mentioned as contributing to this expressive experience. When asked about 
the interdependent networked connections, one melody ball player described 
her experience as a constant state of trying to expect the unexpected. To 
another player, the experience felt like controlling an entity with a life of its 
own. In a manner similar to chamber music group interaction, body and 
facial gestures served an important role in coordinating the accompaniment 
players’ gestures and establishing an effective outcome. Such collaborations 
turned out to be especially compelling for children, who found the accompa- 
niment balls conducive to social interaction, intuitive and easy to play with. 
Some complaints were made, however, regarding the difficulty for individual 
accompaniment players to create their own musical phrases without being 
constantly subjected to interdependent transformation from the group. Other 
criticism addressed the lack of discrete input, which prevented players from 
generating and controlling specific musical events in detail. 
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3.3 The Musical Fireflies (1999-2000) 


The Musical Firefly project was designed to address some of the weak- 
nesses in the Squeezables. In particular, it aimed to facilitate a more discrete 
and autonomous interaction that would allow for clearer interaction schemes 
and more focused constructionist-learning goals. The project attempted to 
provide players with expressive hands-on experiences that can be easily 
transformed into an analytical and formal exploration of music and math- 
ematics. Through simple tapping gestures players could input rhythmic pat- 
terns and embellish them in real-time by adding multiple rhythmic layers. 
This functionality provided players with figural and formal familiarisation 
with musical concepts such as accents, beats, patterns, and timbre. During 
the multi-player interaction, a wireless network was formed between Fireflies, 
which allowed players to synchronise patterns and trade instrument sounds. 
This interactive group experience was designed to lead to deeper internalisa- 
tion of advanced musical concepts such as the correlation between mono- 
rhythmic and polyrhythmic structures. Access to and manipulation of LOGO 
code for customising the controllers provided an introduction to MIDI pro- 
gramming and electronic sound. Advanced players could, therefore, deepen 
their learning experience by reprogramming the controllers and adjusting 
their functionality to match personal musical interests and abilities. 

The 3D printed Musical Firefly’s case was designed to be held by two 
hands while thumb-tapping two top-mounted buttons. Signals from the 
buttons were sent to an embedded “Cricket” Microchip PIC microproces- 
sor. An infrared communication port allowed for communication with other 
Fireflies as well as for downloading LOGO based application programs. The 
played rhythmic patterns were converted 
into musical messages using Cricket 
LOGO general MIDI commands and sent 
through the Cricket’s serial bus port to 
the MidiBoat - a small General Midi cir- 
cuit that supported up to 16 polyphonic 
channels, 128 melodic timbres and 128 
percussive timbres. The audio from 
the MidiBoat was then sent to the top- 


Fig. 3. Two players interact with mounted speaker. 
each other with the Musical Interaction with the Musical Fireflies 


Fireflies mmr Occurred in two distinct and sequential 

modes - the Single Player Mode, where 
players converted numerical patterns into rhythmical structures, and the 
Multi Player Mode, where collaboration with other players enhanced the basic 


rhythmic structures into polyrhythmic compositions (Fig. 3). In Single Player 
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Mode, players could trigger and play with two default percussive sounds. The 
left button triggered accented notes and the right button triggered non-ac- 
cented notes. The patterns of accented and non-accented notes were recorded 
and after two seconds of inactivity, played back in a loop, using an adjust- 
able default tempo. This activity provided players with a tangible manner 
of entering and listening to the rhythmical output of any numerical pattern 
they envisioned, leading to an immediate conceptualisation of the mathemat- 
ical-rhythmical correlation. For example, Figure 4 depicts the playing of the 
numerical pattern 4 3 5 2 2: 


Fig. 4. A pattern of accented and non-accented notes as played by the 
Musical Fireflies. e= Accented note played by the left button; o= non 
accented note played by the right button 


CA 


During playback, players could enter a second layer of accented and non- 
accented notes in real-time, using a different timbre. Each tap on a button 
triggered a note aloud and recorded its quantised position so that the pattern 
became part of the rhythmic loop. Pressing both buttons simultaneously at 
any point stopped the playback and allowed the player to enter a different 
pattern. In Multi Player Mode, when two loop playing Fireflies “saw” each 
other (i.e., when their infrared signals were exchanged), they automatically 
synchronised their rhythmic patterns. (A similar interaction occurs when the 
Firefly insects synchronise their light pulses to communicate in the dark). 
This activity provided participants with a richer, more complex rhythmical 
composition and allowed for an interactive introduction to polyrhythm. Figure 
5 depicts how a 7 beat pattern played by one Firefly and a 4 beat pattern 
played by another diverge and converge as the patterns go in and out of phase 
every 28 beats, the smallest common denominator: 


perar ee eee eee 
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Fig. 5. Two patterns (7/4 and 4/4) played by two Fireflies divergence and 
convergence as they go in and out of phase every 28 beats 
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While the two Fireflies were synchronised, players could also initiate a 
“Timbre Trade” in which instrument sounds were exchanged between the 
devices. Pressing either the left or right button traded both layers of the 
accented or non-accented timbres respectively. Each Firefly continued to 
play its original pattern using the new received timbre. This interaction pro- 
vided players with a higher-level of musical abstraction as they separated the 
rhythmical aspect of the beat from the timbre in which it was played. Because 
the Fireflies network became richer after the interaction (i.e., each instru- 
ment contained four different timbres) the system encouraged collaborative 
play where players were motivated by trading, collecting and playing games 
by sending and receiving different timbres from their peers. 

Observations of play sessions with the Musical Fireflies have been con- 
ducted followed by discussions with the players. Participants were asked 
about the expressive and the educational aspects of the session as well as for 
their suggestions for improvements. A software version of the application was 
prepared and tested. Both novices and experienced users found the concrete 
aspects of playing with a physical object compelling in comparison with the 
graphical user interface of the software version, mentioning the unmediated 
connection that was formed with the instrument as contributing to the crea- 
tion of personal connection with their music they created. Listening to the 
music from distinct physical sources also helped players to follow the inter- 
action in a more coherent manner in comparison to listening to computer 
speakers. The observations and interviews also led to the identification of 
points for improvement and future work. For example, it was clear that the 
focus on a specific constructionist learning activity hampered the open-ended 
expressive gestural interaction goal of the project. Moreover, the simple inter- 
action using only two discrete buttons and the low-quality MIDI sounds led 
to a disappointing musical outcome, consisting mostly of monotonous inter- 
locking clicks with no pitch, time-based rhythmic values, rests, or continu- 
ous transformation. The network interaction in multi-user mode, while effec- 
tive for learning, did not provide a satisfactory collaborative experience. The 
restricted interconnectivity of the system, where discrete timbre-trading was 
the only interpersonal act, did not provide long-lasting rich play value and led 
players to lose interest in the interaction after a few trades. In addition, due 
to the limitations imposed by the line-of-sight infrared communication, the 
application only allowed for synchronisation and timbre trading between two 
players at a time. Many interviewees expressed their wishes to interact and 
collaborate in larger groups comprised of several simultaneous players. 
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3.4 The Beatbugs / “Nerve” (2001-2003) 


For the Beatbug project, new hardware and software applications were 
developed in an effort to address the weaknesses identified in the Musical 
Fireflies. The binary buttons were replaced with a piezo electric sensor that 
could sense hit strength, providing more expressive physical interaction 
through large full-arm drumming gestures. The single user application was 
enhanced to record rhythmic values, rests, pitches, and amplitudes, allow- 
ing for more versatile and expressive musical input. Two new bend sensors 
were added to the design, allowing players to continuously modify and trans- 
form the recorded musical phrases using low- and high-level transforma- 
tion algorithms (Fig. 6). In addition, the embedded MIDIBoat was replaced 
with a high-quality software synthesiser, which significantly enhanced sound 
quality and versatility. Several important enhancements were also made to 
improve the multi-user collaborative interaction. The network was enhanced 
to support up to eight simultaneous Beatbugs, while coloured LEDs were 
installed in each Beatbug to help convey complex multi-user interactions in a 
visual manner. The interpersonal application was improved to provide longer 
lasting collaborative interactions, allowing players to continuously develop 
each other’s music by bending and manipulating the Beatbug antennae. In 
order to support these improvements, the new Beatbugs communicated with 
each other through wires via a central computer system, which was titled the 
“Nerve Center”. To showcase the improved system, a musical composition 
was composed, titled “Nerve”, which was presented in workshops and con- 
certs as part of the Tod Machover’s Toy Symphony project. 

In an effort to provide a familiar and fun interface for children and novices, 
the “Nerve” Beatbug was designed as a bug, having a speaker for a mouth, two 
bend-sensors for antennae, and a velocity-sensitive piezoelectric sensor on its 
back. White and coloured LEDs mounted 
in its translucent shell provided visual 
feedback when hit or played through. An 
embedded Microchip PIC microcontroller 
was responsible for reading input from the 
sensors, controlling the LEDs, and com- 
municating with the central system via 
tail-like cable that carried MIDI, trigger, 
audio, and power. The piezo electric sensor Fig. 6. Manipulating the 


measured when and how hard it was hit, Beatbugs antennae 

while the two antennae allowed for subtle i a ae al 
control over different aspects of the sound. Bending the antennae caused a 
proportional change in the colour of three LED clusters, and a ring of white 


LEDs flashed each time the bug was hit, providing additional visual feedback 
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to the player and audience. The embedded processor was responsible for 
operating the sensors and LEDs, while the central computer system control- 
led the actual musical interactions and behaviours. The “brain” of the system 
was written in Max/MSP environment. Controlling all of the behaviour from 
the central computer made it easy to quickly experiment with a broad range 
of interaction schemes. Similarly, sound synthesis occurring on the central 
computer and played through the corresponding Beatbug’s speaker, provided 
high quality sound with an embedded, self-contained feel. For the software 
synthesiser, ‘Reason’ by Propellerhead was chosen, providing a broad pal- 
ette of timbres and continuous control over multiple sound parameters. Up 
to eight Beatbugs could be connected to one central rack, which consisted 
mostly of standard off-the-shelf equipment including an audio interface, a 
MIDI interfaces, an 8-channel amplifier, and a mixer. The only non-standard 
device in the system was a custom patch box, which provided power to the 
bugs and converted the 10-pin connector in each cable to MIDI in, MIDI out, 
trigger, and audio in (Fig. 7). 
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Cable carries midi, audio, piezo signal 
and power to and from Beatbug 


"Mega" box - routes | drum trigger Drum Module - piezo 
midi, piezo signal, signal to MIDI 
audio, and power 


midi 


MIDI Interface (8 ch) 


audio 
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to 2ch 
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to house mixing console 


Firewire audio 


Audio Interface (8 ch) 


Fig. 7. The Nerve Beatbug system’s schematics 
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Similarly to the Musical Fireflies, players interacted with the “Nerve” 
Beatbug in two distinct modes — Single Player Mode, and Multi Player Mode. 
In Single Player Mode, each player could enter a short rhythmic pattern over 
a predefined metronome beat. The system automatically played back the 
recorded pattern in a loop through the corresponding Beatbug’s speaker. A 
quantisation algorithm pushed the notes towards the closest quarter, eighth 
or triplet note. While the entered pattern was playing back, the player could 
manipulate the pattern by bending the two antennae. The left antenna contin- 
uously transformed the pitch and timbre using a variety of predefined scales 
and audio effects. The right antenna added rhythmic ornamentation to the 
pattern by controlling the values, length, accentuation, and feedback level of 
a delay line. The goal of these transformation algorithms was to allow players 
to modify the pattern but to keep the feel of the original motif, supporting the 
“motif-and-variation” nature of the interaction. In Multi Player Mode players 
could form large-scale collaborative compositions by interdependently shar- 
ing and continuously developing each other’s motifs. Each Beatbug player 
could play a rhythmic motif that was then automatically sent through the 
stochastic computerised “Nerve Center” to another player in the group. The 
receiving player could decide whether to further develop the received motif 
(by continuously manipulating pitch, timbre, and rhythmic elements with 
the two bend sensor antennae) or to keep the motif in his or hers personal 
bug (by entering and sending a newly generated motifs to a different random 
player in the group). The antennae transformations were recorded and layered 
in each cycle until a new pattern was entered. The tension between the sys- 
tem’s stochastic routing scheme and the 
players’ improvised real-time decisions 
led to an interdependent, dynamic, and 
constantly evolving musical outcome. In 
a different section of Multi Player Mode, 
after all players entered their patterns, 
the system awaited a series of simulta- 
neous hits by all players that led to ran- 


dom segmentation of the participants to 
sub-groups, allowing players to interde- Fig. 8. A Beatbug workshop at 
pendently collaborate with a gradually ee ine 
growing number of co-players.? 

During 2002-2003 the “Nerve” Beatbugs were featured in workshops 
and concerts in Berlin, Dublin, Glasgow, Boston and New York in collabora- 
tion with local symphonies and educational programs (Fig. 8). During each 


week-long workshop, children and orchestra members were introduced to 
em 


9 Seea video clip of the interaction as performed in concert at <http: / /www.cc.gatech. 


edu/~gilwein /videos/Glasgow%20-%20Concert.mov>. 


the Beatbugs, explored the system, and rehearsed towards a public concert. 
The workshops also featured a new constructionist pedagogy developed in 
collaboration with Kevin Jennings. The pedagogy was designed to allow play- 
ers to physically create and phrase rhythmic patterns and transform them 
by employing melodic, timbral, and rhythmic contours. The balance among 
aural, kinesthetic and social modalities provided the children with a rich 
and highly immersive musical environment. A report by Project Zero from 
Harvard’s Education School said that “[the project] provided an overwhelm- 
ingly positive experience either from the musical, social and personal stand- 
point... the experience provided a good foundation on which to build one’s 
musicianship, social skills, self-confidence, and general learning dispositions 
focusing, listening, and practicing.” 

Several problems and areas for improvement became apparent as well. 
The musical mappings in Single Player Mode, although more versatile than 
in the Musical Fireflies, were still limited and unsatisfactory for many profi- 
cient musicians, who expressed their interest in creating and manipulating 
more advanced and non-quantised melodic and harmonic musical content. 
Novices too showed interest in controlling more sophisticated musical mate- 
rial even if they could not 
create it themselves. In 
multiplayer interactions, 
the velocity sensing pie- 
zoelectric sensor and the 
large scale of the system 
encouraged players to use 
wide playing gestures and 
expressively point to indi- 
cate their actions to each 
other and to the audi- 


ences (Fig. 9). However, 
while these large gestures 


Fig. 9. Large play gestures in a “Nerve” concert, 
Cambridge, MA 


nn al expression and excite- 


brought elements of vis- 


ment to the performance, 
they were not sensed by the central system and therefore did not have audible 
consequences. In terms of hardware, it was clear that the central system was 
too large and complex, and that the 18-unit rack was not easily portable. An 
additional hardware weakness was the durability of the bend sensor anten- 
nae, which proved to be fragile, especially when large groups of energetic chil- 
dren experimented with the system during week-long workshops. 
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3.5 iltur (2003-2005) 


The iltur project utilised an improved version of the Beatbug controllers, 
which were enhanced both in hardware and software in an effort to address 
the weaknesses observed in Nerve. Hardware improvements included replac- 
ing the unreliable bend sensors with robust Hall effect sensors, installing 
2D accelerometers to sense larger and more expressive arm gestures, and 
reducing the size and complexity of the system. The software was rewritten 
to address users’ requests to control and manipulate advanced melodic and 
harmonic content in a more expressive and gestural manner. The new appli- 
cation supported interaction between Beatbug players and proficient musi- 
cians, allowing Beatbug players to record live input from MIDI and acoustic 
instruments and to respond by transforming the recorded material gesturally, 
creating motif-and-variation call-and-response routines on the fly. The cen- 
tral computer host was programmed to analyse MIDI and audio signals and 
to allow Beatbug players to personalise the analysed material using a variety 
of transformation algorithms. Capturing and personalising richer musical 
content through expressive gestures gave Beatbug players the opportunity to 
create a more sophisticated musical outcome, while forming elaborate musi- 
cal dialogs with their peers. 

The main hardware improvement in the iltur Beatbugs was the addition 
of the 2D accelerometers. The accelerometers were used to sense tilting and 
shaking gestures, providing the central system with information regarding 
players’ large arm movements. Hardware improvements were also made in 
an effort to make the antennae more robust, utilising Hall effect sensors 
and magnets mounted under the antennae. This electromagnetic sensing 
method proved to be robust and effective, although it provided lower bending 
resolution in comparison with the original resistance-based bend sensors. 
Other hardware improvements addressed the system size and portability. As 
opposed to the complex 18-unit rack Nerve system, the new iltur system, 
utilised a laptop instead of a desktop, a software mixer instead of a physical 
one, and no MIDI drum controller, as audio from the piezoelectric sensors 
was captured directly through an audio interface. The system, therefore, was 
housed in a small 6-unit rack (Fig. 10). 

Play gestures and interaction in iltur were modified to allow for record- 
ing, triggering, and manipulation of MIDI and audio in real-time. Recording 
was conducted by simultaneously bending both antennae while tapping the 
Beatbug. The system then segmented the recorded phrases, looking for sec- 
tions of silence in the MIDI and/or audio buffers. The audio Beatbugs were 
programmed to detect onset notes, pitches, and amplitudes in real-time. The 
analysis algorithm was optimised for brass instruments and was used suc- 
cessfully with instruments such as trumpet, trombone and saxophone. Onset 
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Fig. 10. The iltur Beatbug system’s schematics 
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identification and segmentation of MIDI was trivial due to the discrete nature 
of the MIDI protocol. After the system recorded and segmented the captured 
musical input, players could immediately trigger the recorded phrase by tap- 
ping the Beatbug again. Hit velocities were mapped to different segments in 
the phrase, allowing players to rearrange the recorded motifs. Two synthesis 
methods — Wavetable Synthesis and Granular Synthesis — were used for re- 
triggering audio. The Wavetable technique provided close resemblance to the 
sound of original recording but suffered from noise artifacts during continuous 
transformations. Granular Synthesis, on the other hand, provided harsher 
sounds in comparison to the original recording but allowed for smoother con- 
tinuous transformation. A number of different mapping schemes were experi- 
mented with for antennae bending and accelerometer-based gestures. Some 
of these algorithms utilised direct mappings between continuous gestures 
and fundamental musical aspects such as pitch, volume and tempo. Other 
mapping approaches allowed for the manipulation of higher-level musical per- 
cepts such as melodic similarity or rhythmic density. Shaking gestures were 
most successful when mapped to control vibrato and tremolo effects, while 
antennae manipulations were effective in controlling pitch. When interact- 
ing with a MIDI instrument, Beatbug players could also trigger the recorded 
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motif in inversion and retrograde by tapping the Beatbug while bending the 
left or right antennae, respectively. The audio Beatbugs allowed players to 
control transformations such as pitch bending, speed alteration, and filtra- 
tion, through a combination of bending, tilting, and hitting gestures. During 
group interaction, players could trade their motifs by simultaneously hit- 
ting the Beatbug while bending one of the antennae. Receiving players could 
then further transform 
the phrase and send it 
back to their peers. In 
comparison to the ran- 
dom involuntarily rout- 
ing scheme in Nerve, 
iltur players could trade 
their motifs only when 
simultaneously agreeing 
to synchronise their ges- 
tures. Three Jazz com- 


positions were written 
for the iltur system and 


Fig. 11. iltur 3 audio Beatbug players interact 
with a brass section (left) and a hip-hop vocalist 
(right) in Jerusalem, Israel 
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performed in cities such 
as Atlanta, San Diego, 
Miami, Vancouver, and 
Jerusalem. iltur 1 featured MIDI interaction, iltur 2 focused on audio transfor- 
mation and manipulation, and iltur 3 introduced group interaction and motif 
trading. Voice manipulation experimentations were also conducted, allowing 
Beatbug players to interact with a hip-hop vocalist.!° 

Observations of and discussion with iltur players led to a number of find- 
ings regarding the improved Beatbug functionalities. For example, it was clear 
the iltur Beatbugs were more effective than the Nerve Beatbugs in providing 
richer musical experiences for individuals through a larger set of expressive 
gestures and more complex melodic and harmonic transformations. The new 
application also led to more meaningful and versatile collaborations between 
novices and professional musicians. Both players and audiences perceived 
the new accelerometer-based gestures as intuitive, expressive, and visually 
compelling. However, the introduction of gesture combinations (such as hit- 
ting the Beatbug while bending the antenna) was problematic for novices and 
children, who found it physically and mentally challenging. Novices and chil- 
dren also found the higher-level transformation algorithms (such as musical 
density and stability) less intuitive to control and preferred the simple and 
predictable one-to-one mappings between gestures and low-level musical 
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10 See videos at <http: / /www.cc.gatech.edu/~gilwein/iltur.htm>. 


315 


aspects. More proficient musicians, on the other hand, preferred to interact 
with the high-level musical operations, stating that these encouraged them to 
concentrate on the correlation between their actions and the musical output. 
In general, the effective- 
ness of the experience 
was closely related to the 
musical and harmonic 
context of the composi- 
tions. Due to segmenta- 
tion and audio stretch- 
ing, in a harmonically 
structured composition 
it was difficult for play- 
ers to improvise while 


following the harmonic 
Fig. 12. Interaction between two iltur 3 MIDI progression. Many play- 
Beatbug players ers, therefore, preferred 
ALAA 

free musical structures, 
stating that open-ended experience posed less boundaries and allowed more 


creativity and expression. 


3.6 Haile (2004-2007) 


The instruments and controllers discussed above explored different ways 
in which meaningful embodiment of technology can enhance the musical 
experience by facilitating new expressive gestures, networked group collabo- 
rations and constructionist learning. Although these projects provided satis- 
fying results, the instruments were limited by the electronic reproduction and 
amplification of sound through speakers, which did capture the richness of 
acoustic sound. My most recent project — an interactive robotic percussionist 
named Haile — addressed this limitation by utilising a mechanical apparatus 
that converts digital musical instructions into acoustic and physical genera- 
tion of sound. Haile was developed in an effort to bring together the advan- 
tages of computational power with the expression and richness of creating 
acoustic sound using physical and visual gestures. 

The project aimed to combine that are not possible by humans with rich 
sound and visual gestures that cannot be reproduced by speakers in an effort 
to facilitate new musical experiences, and new music, that cannot be con- 
ceived by acoustic or means. 

As part of the project, a robotic percussionist that listened to and ana- 
lysed live musical input in real-time and reacted by generating relevant, but 
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at times surprising, acoustic responses was developed. The project posed 
challenges in areas such as perception modeling, mechanics, and interaction 
design. In perception, the main challenge was to implement models for low- 
and high-level musical percepts, allowing the robot to develop a meaningful 
representation of the music it listened to. In mechanics the challenge was 
to develop a dexterous robotic apparatus that would translate perceptually 
based performance algorithms into a rich acoustic and visually informative 
performance. In interaction design, our aim was to develop performance algo- 
rithms that would enable the robot to collaborate with human players in a 
meaningful and intuitive manner, using transformative and generative meth- 
ods both sequentially and synchronously. 

In order to support familiar interactions with human players, Haile’s 
design is anthropomorphic, utilising two percussive arms that can move to 
different locations and strike with varying velocities (Fig. 13). The first pro- 
totype was designed to play a Native American Pow Wow drum - a multi 
player instrument that supported the collaborative nature of the project. For 
pitch-oriented applications, the robot was later adjusted to play a one-octave 
xylophone. In order to match the aesthetics of these musical instruments, 
Haile was constructed from wood using a CnC cutting machine. Metal joints 
were designed to allow shoulder and elbow 
movement as well as leg adjustability for dif- 
ferent instrument heights. While attempting 
to create an organic look for the robot, it was 
also important that the technology was not 
completely hidden, so that co-players could 
see and understand the robot’s operation. 
The mechanical apparatus was therefore 
left uncovered and LEDs were embedded on 
Haile’s body, providing an additional repre- 
sentation of the mechanical actions. Haile’s 
right arm was designed to play fast notes, 
while the left arm was designed to produce 


larger and more visible motions that pro- 
duce louder sounds. Both arms could adjust 


Fig. 13. Haile, the percep- 
tual robotic percussionist, 


listens to and interacts with 
ments in different locations, and volume was a human player 


the strikes sound in two manners: different 
pitches were achieved by striking the instru- 
adjusted by hitting with varying velocities. “OOOO 
To move to different vertical positions, each 
arm employed a linear slide, a belt, a pulley system, and a potentiometer to 
provide feedback. Unlike robotic drumming systems that allow hits at only a 
few discrete locations, Haile’s arms moved continuously over a distance of 10 
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inches (movement timing is 250 ms. from end to end). The right arm’s strik- 
ing mechanism was loosely based on a piano hammer action and consisted 
of a solenoid driven device and a return spring. The right arm stroked at a 
maximum speed of 15 Hz, faster than the left arm’s maximum speed of 11 
Hz. However, the right arm did generate a wide dynamic range or provided 
easily noticeable visual cues, which limited Haile’s expression and interac- 
tion potential. The left arm was designed to address these shortcomings, 
using larger visual movements, and a more powerful and sophisticated hit- 
ting mechanism. 

The first phase of the project aimed at facilitating rhythmic collaboration 
between human drummers and Haile, addressing aspects such as rhythmic 
perception, improvisation, and interaction design. Perceptual models were 
developed for low- and high-level rhythmic percepts, from beat and density 
analysis, to rhythmic stability and similarity perception. Some relatively low- 
level perceptual modules included beat analysis, where domain detection was 
followed by autocorrelation of tempo and phase, and density analysis, where 
we looked at the number of note onsets per time unit to represent the den- 
sity of the rhythmic structure. Higher-level rhythmic analysis modules were 
also developed for percepts such as rhythmic stability, based on research by 
Desain, et al.!!, and rhythmic similarity based on Tanguiane’s survey'?. The 
stability model calculated the relationship between pairs of adjacent note 
durations, rated according to their perceptual expectancy based on three 
main criteria: perfect integer relationships were favoured, ratios had inherent 
expectancies (i.e., 1:2 was favoured to 1:3 and 3:1 was favoured to 1:3), and 
durations of 0.6 seconds were preferred. The similarity rating was derived 
from Tanguiane’s binary representation, where two rhythms are first quan- 
tised, and then given a score based on the number of note onset overlaps and 
near-overlaps. 

The main challenge in designing the rhythmic interaction with Haile was 
to implement the perceptual modules in a manner that would lead to an 
inspiring human-machine collaboration. The approach taken was based on 
a theory of interdependent group interaction in interconnected musical net- 
works. At the core of this theory is a categorisation of collaborative musical 
interactions in networks of artificial and live musicians based on sequen- 
tial and synchronous operations with centralised and decentralised control 
schemes. Based on this framework, six interaction modes were developed: 
Imitation, Stochastic Transformation, Perceptual Transformation, Beat 
Detection, Simple Accompaniment, and Perceptual Accompaniment. These 
interaction modes utilised different perceptual modules and were embedded 
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in different combinations in interactive compositions and educational activi- 
ties. In the first mode, Imitation, Haile merely repeated what it heard based on 
its low-level onset, pitch, and amplitude perception modules. Players could 
play a rhythm and after a couple of seconds of inactivity Haile imitated it in 
a sequential call-and-response manner. Haile used one of the arms to play 
lower pitches close to the drumhead centre and the other arm to play higher 
pitches close to the rim. In the second mode, Stochastic Transformation, 
Haile improvised in a call-and-response manner based on players’ input. 
Here, the robot stochastically divided, multiplied, or skipped certain beats in 
the input rhythm, creating variations of users’ rhythmic motifs while keeping 
their original feel. Different transformation coefficients were adjusted manu- 
ally or automatically to control the level of similarity between humans’ motifs 
and Haile’s responses. In the Perceptual Transformation mode, Haile ana- 
lysed the stability level of users’ rhythms, and responded by choosing and 
playing other rhythms that had similar levels of stability to the original input. 
In this mode Haile automatically responded after a specified phrase length. 
Imitation, Stochastic Transformation, and Perceptual Transformation were 
all sequential interaction modes that formed decentralised call-and-response 
routines between human players and the robot. Beat Detection and Simple 
Accompaniment modes, on the other hand, allowed synchronous interaction 
where humans played simultaneously with Haile. In Beat Detection mode, 
Haile tracked the tempo and beat of the input rhythm using complex domain 
detection function and autocorrelation, which led to continuously refined 
assumptions of tempo and phase. A simpler, yet effective, synchronous inter- 
action mode was Simple Accompaniment, where Haile played pre-recorded 
MIDI files so that players could interact with it by playing their own rhythms 
or by modifying elements such as drumhead pressure to modulate and trans- 
form Haile’s timbres in real-time. This synchronous centralised mode allowed 
composers to feature their structured compositions in a manner that was 
not susceptible to algorithmic transformation or significant user input. The 
Simple Accompaniment mode was also useful for sections of synchronised 
unisons where human players and Haile played together. Perhaps the most 
advanced mode of interaction was the Perceptual Accompaniment mode, 
which combined synchronous, sequential, centralised and decentralised 
operations. Here, Haile played simultaneously with human players while lis- 
tening to and analysing their input. It then created local call-and-response 
interactions with different players, based on its perceptual analysis. In this 
mode amplitude and density perceptual modules were utilised — while Haile 
played short looped sequences (captured during the Imitation and Stochastic 
Transformation modes) it also listened to and analysed the amplitude and 
density curves of human playing. It then modified its looped sequence, based 
on the amplitude and density coefficients of the human players. When the 
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rhythmic input from human players was dense, Haile played sparsely, provid- 
ing only the strong beats and allowing humans to perform denser solos. When 
humans played sparsely, on the other hand, Haile improvised using dense 
rhythms that were based on stochastic and perceptual transformations. Haile 
also responded in direct relationship to the amplitude of human players so 
that the louder humans played, the stronger Haile played to accommodate 
the human dynamics, and vice versa.!? 

As a creative outcome for these interactive applications, two compositions 
were written for the system, each utilised a different set of perceptual and 
interaction modules. The first composition, titled Pow, was written for one or 
two human players and a one-armed robotic percussionist. It served as test 
case for Haile’s early mechani- 
cal, perceptual, and interaction 
modules. The second composi- 
tion, titled Jam’aa (“gathering” 
in Arabic), built on the unique 
communal nature of the Middle 
Eastern percussion ensem- 
ble, attempting to enrich its 
improvisational nature, call- 
and-response routines, and 


virtuoso solos with algorithmic 


transformation and human- 


Fig. 14. A performance of Jam’aa in 
Odense, Denmark 


INN Jam’aa, was commissioned 


robotic interactions (Fig. 14). 


by Hamaabada Art Centre In 
Jerusalem, and later performed in invited and juried concerts in France, 
Germany, Denmark, and the United States.'* 

As part of our effort to expand the exploration of robotic musicianship 
into pitch and melody, Haile was later adapted to play a pitch-based mallet 
instrument. A one-octave xylophone was built for this purpose to accommo- 
date Haile’s mechanical design - the left arm covered a range of 5 keys while 
the right arm, whose vertical range was extendable, covered a range of 7 keys. 
(Fig. 15). Following the idiom “listen like a human, improvise like a machine”, 
computational models for melodic similarity were developed (“listen like a 
human’) as the fit function of a genetic algorithm based improvisation engine 
(“improvise like a machine”). The algorithmic responses were based on the 
analysed input as well as on internalised knowledge of contextually relevant 


13 See a video excerpts of some of the interaction modes at <http:/ /www.cc.gatech. 
edu/~gilwein/Haile.htm>. 


14 See a video excerpts from Jam’aa at <http://coa.gatech.edu/~gil/RoboraveShort. 
mov>. 
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material. The algorithm fragmented 
MIDI and audio input to short phrases. 
It then attempted to find a “fit” response 
by evolving a pre-stored human-gener- 
ated population of phrases using a vari- 
ety of mutation and crossover functions 
over a variable number of generations. 
At each generation, the evolved phrases 
were evaluated by a fitness function that 
measured similarity to the input phrase, 
and the least fit phrases in the database 


Fig. 15. Haile’s adaptation for 
xylophone 


a2 zZ 


are replaced by members of the next generation. A unique aspect in this 


design was the reliance on a pre-recorded human-generated phrase set that 


evolved over a limited number of generations. This allowed musical elements 


from the original phrases to mix with elements of real-time input to create 


hybrid, and at times unpredictable, responses for each given input melody. 


Two compositions were written for the system and performed in concerts in 


Fig. 16. A performance of Svobod in 
Copenhagen, Denmark 


ZA 


ZA 


Atlanta and Copenhagen. In the 
first piece, titled “Svobod”, a 
piano and a saxophone player 
freely improvised with a semi- 
autonomous robot (Fig. 16). The 
titled “iltur for 
Haile”, involved a tonal musi- 


second piece, 


cal structure utilising geneti- 
cally driven and non-genetically 
driven interaction schemes, as 
the robot performed autono- 
mously with a jazz quartet. !* 


15 See a video clip of iltur for Haile at <http:/ /www.coa.gatech.edu/-gil/iltur4haile. 


mov>. 
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1. Introduction 


The rapid evolution of the computer in the 90s introduced further devel- 
opments in Artificial Intelligence, Artificial Life, and Virtual Reality, not to 
mention advances in multimedia and internet technology which are now 
an essential part of our lives. The people who have applied these technolo- 
gies, are not only researchers, but also artists whose work could potentially 
change the way art is viewed. During the interplay of art and digital technol- 
ogy, many new amalgamated fields in the world of art have arisen: Interactive 
Music, Interactive Installations combining sound and imagery, Performance 
Art, Interactive Imagery etc. 

Since the 1990s, I experimented with numerous compositions and per- 
formances using new technologies. Besides compositional concerns, my work 
has been based on questions of whether we are able to further develop the 
aesthetics that have been built by various artists in these different fields 
using interactive computer technology, and to create a new field of which the 
robot is the point of contact. 

This paper is intended to cover my recent works: the virtual violin 
“SuperPolm” (1996), the “BodySuit” (since 1997), “RoboticMusic” (2003) and 
the project “Augmented Body and Virtual Body” (since 2002), both from the 
technical and aesthetic points of view. 


2. “SuperPolm” 


The Virtual Violin “SuperPolm” which was created with the collaboration 
of Patrice Pierrot and Alain Terrier in IRCAM, France in 1996, is one of the 
Virtual Musical Instruments that I created. Virtual Musical Instruments, 
which are defined as systems containing gesture, gesture interface, mapping 
interface, algorithm, and sound synthesis, consist of a gesture interface or 
controller, which cannot produce sounds by itself (Fig. 1). 
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Fig. 1. Components of Virtual Musical Instruments 


0 es 0 


ZZ 0200220220200 EZ 


1 Goto 2000, p. 220 


rar é 
ef s 
S S g F 
EE SF ‘ 

S S 

< 
SF or 
fe s S 
eg < r 
& S 


It merely sends signals that produce sounds by means of a computer or a 
sound module. It may be regarded as an interface between the performer and 
the computer insofar as it translates the energy derived from body movements 
into electrical signals. At the same time however, it allows the performer to 
express complex musical ideas. With the capabilities of gestural interfaces of 
a controller, which can be modified by programming, a tiny gesture can trig- 
ger any number of complex musical passages at one and the same time in a 
real time context, whereas a traditional instrument can produce only a lim- 
ited range of sounds. I use this principle of gesture interfaces as an essential 
factor in my compositions. One of my gestures at one moment might produce 
a sound similar to a traditional instrument but in the following section the 
same gesture might trigger a very different sound. As well as allowing for 
more possibilities in terms of sound, it also allows for a certain theatricality in 
performance. A controller is adapted to a specific type of gesture. In this case 
a controller refers to the gestural interface, but it also means a remote control 
device for manipulating a computer from a distance through MIDI (Musical 
Instrument Digital Interface), OSC (Open Sound Control) etc.? 

The basic idea behind the SuperPolm 
is to interface gestures that resemble the 
playing of a musical instrument - in this 
case a violin — in order to control sound 
or images. These gestures are translated 
into parameters of position, pressure or 
distance by sensors based on the idea 


of short-range motion capture, such 


as finger, hand and arm movements. 
Fig. 2. The Virtual Violin, the 8 


“SuperPolm,” is based upon the 
idea of short-range motion cap- analogue-to-digital interface into MIDI 


ture, such as finger, hand and signals that can be fed into a computer. 
arm movements. 


BZ 2 za 


The resulting voltage is converted by an 


The computer controls or generates the 
sounds in real time and can modify these 
signals by means of algorithms. For example, a single channel signal can be 
altered to become a rich and complex sound such as that of an orchestra 
(Fig. 2). The SuperPolm contains a force sensor placed in the chin rest and an 
inclinometer measuring respectively the performer’s constraint to maintain 
the instrument and the angle impressed towards the vertical. Therefore the 
performer can control two added parameters without hand movements using 
chin pressure and/or bending the upper body forwards. 

The SuperPolm can be played in a similar manner to the violin, except 
that the fingers touch sensors on a fingerboard instead of pressing strings, 
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since there is neither string nor hair of bow: A gesture of performance with 
a violin is merely modeled (Fig. 2). However, movements of the bow causing 
variations in resistance can, assign new functions as well as modify sounds. 
An eight-button keyboard situated on the body of the instrument can change 
both the program in Max/MSP/Jitter and the sounds, as well as triggering 
different pitches, like a normal keyboard. Hence, new functions of program- 
ming can be taken into account according to the compositional needs of each 
piece: for instance a sensor can be used to trigger sounds in one composition, 
whereas in another it can be used to change the pitch. 

The SuperPolm was originally intended for use in a piece I composed at 
IRCAM in 1995-1996, entitled “VirtualAERI”. The first performance of this 
piece was given in 1997 at IRCAM’s Espace de Projection. It consisted of four 
sections, each of which dealt with a different kind of space, large, medium 
and small. The SuperPolm was designed for one particular section of this 
composition focusing specifically on the 
possibilities opened up by the controller. 

The SuperPolm can control not only 
the parameters of sound synthesis, but 
also those of images in real time (Fig. 3). 
For instance, it can superimpose live or 
sampled images on top of each other, 


add effects, such as delay, and speed up, 
reverse or repeat these images. It can also 


Fig. 3. The SuperPolm can 


mix several images in different propor- also control the parameters of 
tions and modify their colour, brightness images in interactive videos 
and distortion, while the sampled images (Photo and Copyright: Arianna 
can be started and stopped at any point. fc Keen EEE 


3. “BodySuit” 


In my projects, Virtual Musical Instruments have been used in a per- 
formance context. Another instrument I have designed is the “BodySuit”, a 
suit fitted with bending sensors that are attached to each joint of the body. 
It was built between 1997 and 1999 with the aim of a motion capture for the 
entire body, so that a performer wearing it can merely produce sounds with 
his gestures by bending and stretching each joint, without controlling any 
instruments in his hands. 

The “BodySuit” is equipped with 12 sensors which are attached on each 
joint of the body. Depending on a movement, sound and video images are 
changed in real time. This differs from a traditional instrument or an instru- 
ment-inspired controller. A player performs with larger movements, such as 
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stretching and bending joints, twisting arms and so on. This gesture does not 
function like dance or theater. It contains, however, an element of “perform- 
ance” within the live, musical context. The gesture is not previously decided in 
a strict sense. An audience may observe 
an obvious difference of intensity of 
movement between a static section and 
a kinetic section in the composition (Fig. 
4). This suit is therefore an ideal tool in 
a musical theater situation. 

Although the performer’s ges- 
ture does not resemble those used for 


playing a musical instrument, I used 
the BodySuit exclusively as a virtual 


Fig. 4. The BodySuit enables to 
make wide, sweeping movements 
that can easily be observed by works efficiently with percussionist-like 
the audience. Performance at gestures, as designed for my project 


the University of Cologne in April “RoboticMusic” which will be described 
2004 (Photo: Franca Lohmann) 


AMAA in section 3 in detail. 


musical instrument. In particular, this 


4. “RoboticMusic” 


The act of performing music is not only about the control of a complicated 
set of body movements. On the one hand, music can be seen as a logical 
sequence of events over time which occur as a result of problem solving and 
rely on the interaction of a set of parameters. On the other hand, music 
derives itself from less calculable things such as ‘good’ rhythm sense and 
poetic significance and expression. 

Taking into account such aspects of music performance, I was interested 
in the question of whether it would be possible for a robot making music to 
think logically, to play with emotion, to have a good sense of rhythm, to realise 
poetic expression, to achieve proper pitch (frequency) and delivery of sound, 
and to have a sense of proportion through comparison. Therefore I have been 
engaged in projects using robots in music performances, especially robots 
playing musical instruments. 

My project “RoboticMusic” was designed for robots that consist of Snare 
Drum, Bass Drum, Cymbal, Gong, and Pipe. These have a specially designed 
springs to imitate human muscles. Each holds a mallet at the end of its arm 
(Fig. 5). One of the robots plays numerous pipes, and rapidly spins to cre- 
ate Flute-like sounds, which are generated as the air goes through them. 
These pipes are different lengths according to the pitches one desires. As it 
spins faster, the pitches become higher, moving up the overtone series. These 
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mechanisms are created by me with technical help from Fuminori Yamazaki, 
iXs Research Corporation in 2003. 

These robots performing on musical instruments are connected by com- 
puter and controlled with a program. Max from Cycling’74 is utilised both as 
an interface and to generate musical data. With this, one can also send basic 
parameters to the robots such as a position of the robot’s arm, an offset posi- 
tion, intensity (how hard it hits) and so on. This sends the signals to another 
computer running Linux via UDP (Universal Data Protocol). This software in 
Linux is developed by iXs Research Corporation. This plays an important role 
since it controls the robot’s movement. The 
Linux computer and robots are connected 
via USB (Universal Serial Bus). Each robot 
has its own interface which is connected 


| -- 


with an actuator and a sensor. 

The major advantage of “RoboticMusic” 
is that it interactively plays an acoustic 
instrument with the aide of a computer. 
There is no problem with playing complex ie | 


rhythms which easily outperform human Fig. 5. Each holds a mallet at 


capabilities. Therefore, it gives new poten- the end of his arm and can 
tialities in composition for acoustic instru- play any instruments as long 
ments. While a computer generated sound as these can be played with 
has many capabilities, an acoustic instru- Een 


ment has rich sonority and enormous pos- 

sibilities of expression, especially from the composer’s point of view. With 
staged peformance the vast possibilities of the acoustic aspect are obvious 
when compared to sound coming from speakers. Another benefit is that the 
audience may observe both the source of the sound and the accompanying 
gestures necessary for its production. 

I explored some musical compositions in order to see what only these 
robots could play. For instance, five robots in the project “RoboticMusic” 
played on musical instruments with different tempos at the same time, or 
intricate accelerando and ralentando, and yet synchronisation was main- 
tained during the playing process. Each algorithm is assigned to each robot, 
but these 5 algorithms are controlled by one central program. 

Other possibilities allow the robots to improvise and compose by them- 
selves in real time during their performances with the aide of the computer’s 
algorithm. During these performances, they sometimes played with compu- 
ter-generated sounds at the same time. While I played a laptop, the robots 
accompanied me. In these performances, the lights were much emphasised, 
as well. As the robots changed their performances, the automated lights 
communicated with them via MIDI. The music, gestures on stage, and the 
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visual elements of the lights were 
integrated into one whole stage per- 
formance in RoboticMusic (Fig. 6). 


5. “Augmented Body 
and Virtual Body” 


This project “Augmented Body 
and Virtual Body” originally started 
in 2002. The system used for this 
project consists of the Data Suit, 


Fig. 6. Especially, these robots can 
show a lot of potentialities in con- 


certs. The robots can certainly per- “BodySuit”, and the Percussion 
form faster and more accurately than Robots, “RoboticMusic”, controlled 
human players. in real-time by gestures produced 
EEE 


with “BodySuit”. This system 
was intensively experimented with and shown on several occasions during 


2005. The last performance was realised in, “Le Cube,” in France in April 
2006. The idea behind the system is that a human body is augmented by 
electronic signals in order to be able to perform musical instruments inter- 
actively. A gesture of a performer with “BodySuit” is translated to gestures 
of “RoboticMusic.” One of the important elements is the relationship and 
the communication method explored within this system. One may consider 
“BodySuit” and “RoboticMusic” as a relationship between a conductor and an 
orchestra, where dance-like gestures trigger instruments. In other words, this 
is an instrument that relies on physical gestures. Another point is the method 
of translation used by the computer. For example, signals from “BodySuit” are 
transformed by mapping interface and algorithm in a computer, and routing 
them into “RoboticMusic.” One gesture may trigger one attack on one instru- 
ment. However, it is also possible to trigger five instruments at the same 
time. Otherwise complex, musical data, which is automatically generated by 
a computer and then reproduced by “RoboticMusic”, is altered with gestures 
from “BodySuit” to modify the parameters of an algorithm in real time.* 

The robots’ reactions to the “BodySuit” performer’s gestures can be either 
direct or indirect. For example, a rapid arm gesture from a higher position to 
lower could trigger the percussion robot to hit an instrument, or a gesture 
triggers an algorithm of particular behaviour that sends signals with various 
values of delay to each robot. The robots which are controlled by the compu- 
ter, only play in certain sections of the composition. While the passages which 
are controlled by “BodySuit” make it possible to create a more complex musi- 
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cal material. The robot solo with the computer allows realisation of complex, 
high speed performance, which is impossible for human instrumentalists. 

While the communications of gesture with “BodySuit” and “RoboticMusic” 
are observed, one notices different phases of interaction, which are the inter- 
action with its perception, and the interaction with its consciousness. These 
two poles are important keys in this field. With the articulated visual and oral 
experience in this work, one may recognise different experiences that con- 
stantly deal with something to expect, to understand, to notice, and to per- 
ceive. Furthermore, the relationship between gesture and sound is also seen 
differently with this system. In other 
words, the idea of, “music to see, vis- 
ibility to hear” brings a different context 
into theatrical performance. 

Let us see the relationship between 
“BodySuit” and “RoboticMusic”. The first 
is designed to control others with the will 
of a performer. The latter is designed to 


be controlled by someone else. However, 
both of these equate to bodies that are 
extended with the medium of electronic Fig. 7. The BodySuit works well 
with _percussionist-like ges- 
tures. This is one of the best 


controller conjunctions with 
mented, are intentionally mixed. On the “RoboticMusic”. 


devices. The meaning of body, which 
can exist as being virtual or being aug- 
other hand, when we see the relation- (20; 
ship between “BodySuit” and “RoboticMusic”, this appears as a relationship 
between the physical world and the virtual world. While the robots consist 
of artificial bodies compared with human bodies, they could also be physical 
bodies contrasted with virtual bodies. 


6. Conclusion 


The idea underlying my projects sketched above is to explore the dual- 
ism of the real and the virtual and the relationship between artificiality and 
reality of the human body. Artificiality and reality sometimes seem to be in 
conflict with each other, but they can work together, or their meaning can be 
transformed for an audience depending on the context. The context provokes 
the audience into playing with the ideas of reality and artificiality. A perform- 
ance involving “BodySuit”, and “RoboticMusic” challenges the audience by 
confusing the line between virtuality and reality. As a composer I intend to 
create such a composition which emphasises the importance of performance 
aspects with this system. 
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While the concept of “Extended Body” was conceived to be realised with 
these systems mentioned above, the theme “Augmented Body and Virtual 
Body” is meant to question what a human body is and what its own identity 
is with this. Man and Machine seem to be dualistic, more precisely, one may 
think that they are conflicted against each other. In my projects, however, they 
coexist within an interactive, artistic system: man and machine are regarded 
as being one — an “Extended Body”. As a result, our identity is not merely 
confined within the boundaries set up by our body, but becomes extended. 
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